Tutorial index: Big Data for Beginners: tutorial series
Having finished the Hadoop beginner series, we’re finally getting to the real data warehouse work. Let’s start by understanding what Hive is in the Hadoop ecosystem.
What Is Hive?
Hive is a data warehouse built on Apache Hadoop. Facebook open sourced it to handle statistics over massive volumes of structured logs. It manages and queries the data living in Hadoop.
Hive was designed to make aggregation, querying, and analysis simpler over huge datasets. It provides SQL so users can query, aggregate, and analyze data more easily, and its SQL offers several ways to plug in your own functionality for customized queries.
The way I see it, Hive is essentially a SQL parsing engine: it translates SQL statements into MapReduce jobs for execution, which makes working with the data much easier. Because it’s built on MapReduce, Hive is not designed for online transaction processing. It fits traditional data warehousing tasks best.
As noted above, Hive is really built on MapReduce, so it maps SQL tables and fields to files (directories) in HDFS and columns within those files. That mapping produces mapping data — metadata — called the metastore, usually stored in Derby or MySQL.
Given all that, I think of Hive as a high-level wrapper around MapReduce, much like MyBatis wraps JDBC to make it easier to use.
Compared with Traditional Databases
The introduction called Hive a data warehouse rather than a database, yet both speak SQL. So what’s actually different? Quite a lot, as it turns out.
Query Language
Hive ships HQL, a SQL-like language designed around its own characteristics — it is not the SQL you already know, exactly.
Where Data Lives
Hive sits on top of Hadoop, so all Hive data is stored in HDFS. Databases can keep data on block devices or in the local filesystem.
Data Updates
The earlier tutorials explained that HDFS handles data rewriting poorly, so Hive discourages modifying data: everything is fixed at load time. If your data changes constantly, Hive isn’t for you.
Indexes
Hive has no indexes. Loading data performs no processing on it — not even a scan — so no indexes get built over any key. When Hive needs rows matching a condition, it brute-force scans the entire dataset, which means high access latency.
Summary
To sum up: Hive is a data warehouse suited to non-real-time access, poor at data mutation, and slower to execute than a database — it simply converts SQL into MapReduce. What it buys you is HDFS’s ability to store enormous volumes of data. Next post, we’ll install Hive.