29 posts
The original post derived two complements, CuB and CuA, from the first comparison, then pulled the real data for a second comparison. Applying this in production revealed that pulling the source data in the second comparison can be skipped, so I've corrected the ETL steps from that post.
The previous big chapter covered the concept of a big data warehouse and how to think about building one. Now we start turning those ideas into reality. The foundation carrying all of it is the collection of big data components in the Hadoop ecosystem, which gradually becomes our data warehouse and platform.
Last post hashed the digest of every field in each row, compressing dozens of fields into a single hash to speed up comparison. But that only compressed the fields — the row count stayed the same. With billions of rows we still can't find the changed records fast enough.
Last post covered fact tables, dimension tables, the star schema and the snowflake schema. Beyond those there's more industry jargon to learn. This post explains the big data slang: full tables, incremental tables, zipper tables, transaction tables, and snapshot tables. Some of it may not click yet — it will once you work with Hive — so take this as foundational knowledge.
In the previous post on data warehouse layering we mentioned that each layer does some processing beyond the raw tables, and that processing involves fact tables and dimension tables. This post gives you a rough understanding of fact tables, dimension tables, and the star and snowflake data models.
As data flows into our warehouse over time, both the variety and volume grow. Without governance and design, querying and retrieving that data becomes very painful — so the warehouse needs designing, with each type of data filed where it belongs, ready to be called up whenever we need it.
This is the opening post of my big data beginner series. Every big data component and tool we use exists to solve data problems, and data has to be stored and retrieved through a data warehouse — no technical architecture escapes that. So let's start by understanding what a data warehouse is, and how it differs from a database.
Covers data warehouse concepts; beginner tutorials for Hadoop, ZooKeeper, Hive, Flume, Kafka, Hbase, Sqoop, Oozie, Azkaban, Kylin, CDH, Impala, Hue, ClickHouse, Kettle, Ambari, ELK, Scala and Flink.
You can't talk about data warehousing without talking about ETL. The term is most common in warehouses, but it isn't limited to them. It's a critical step, so here's a quick introduction to how ETL extraction and comparison work.
