219 posts
The original post derived two complements, CuB and CuA, from the first comparison, then pulled the real data for a second comparison. Applying this in production revealed that pulling the source data in the second comparison can be skipped, so I've corrected the ETL steps from that post.
MariaDB/MySQL is case-sensitive on Linux, so any mismatch in capitalization between your table or column names and your SQL throws errors. That's why we usually turn case sensitivity off first.
The previous big chapter covered the concept of a big data warehouse and how to think about building one. Now we start turning those ideas into reality. The foundation carrying all of it is the collection of big data components in the Hadoop ecosystem, which gradually becomes our data warehouse and platform.
In SQL, the LIKE operator searches for a specified pattern in a column inside a WHERE clause. It's a very common way to query in practice, and most people only use the % wildcard — but several other patterns are supported too.
Last post hashed the digest of every field in each row, compressing dozens of fields into a single hash to speed up comparison. But that only compressed the fields — the row count stayed the same. With billions of rows we still can't find the changed records fast enough.
Last post covered fact tables, dimension tables, the star schema and the snowflake schema. Beyond those there's more industry jargon to learn. This post explains the big data slang: full tables, incremental tables, zipper tables, transaction tables, and snapshot tables. Some of it may not click yet — it will once you work with Hive — so take this as foundational knowledge.
This is a decade-old vulnerability in the sudo tool that can grant any local user root access on Unix-based systems, including macOS Big Sur and earlier. Improper operation grants root privileges to arbitrary local users. To trigger it, a user only needs to rewrite argv[0] or create a symbolic link.
In the previous post on data warehouse layering we mentioned that each layer does some processing beyond the raw tables, and that processing involves fact tables and dimension tables. This post gives you a rough understanding of fact tables, dimension tables, and the star and snowflake data models.
As data flows into our warehouse over time, both the variety and volume grow. Without governance and design, querying and retrieving that data becomes very painful — so the warehouse needs designing, with each type of data filed where it belongs, ready to be called up whenever we need it.
This is the opening post of my big data beginner series. Every big data component and tool we use exists to solve data problems, and data has to be stored and retrieved through a data warehouse — no technical architecture escapes that. So let's start by understanding what a data warehouse is, and how it differs from a database.
