Tutorial index: Big Data for Beginners: tutorial series
This is the opening post of my big data beginner series. Every big data component and tool we use exists to solve data problems, and data has to be stored and retrieved through a data warehouse — no technical architecture escapes that. So let’s start by understanding what a data warehouse is, and how it differs from a database.
RenFei’s note: this post assumes you already know traditional structured databases, so I compare the data warehouse directly against a database. If you’re unfamiliar with structured databases, learn those basics first.
What Is a Data Warehouse?
A data warehouse is commonly abbreviated DW or DWH — remember the abbreviation, because the industry and later posts use DW constantly. Rather than take my word for it, here’s Wikipedia’s description:
A data warehouse is a system used for reporting and data analysis and is considered a core component of business intelligence. Data warehouses are central repositories of integrated data from one or more disparate sources. They store current and historical data together, used for creating analytical reports for workers throughout the enterprise.
The definition proposed by Bill Inmon, the father of data warehousing, in his 1991 book Building the Data Warehouse is widely accepted: a data warehouse is a subject-oriented, integrated, non-volatile, time-variant collection of data used to support management decision making.
From Inmon’s description we get the characteristics of a data warehouse:
- Subject-oriented: data is organized around specific subject areas.
- Integrated: data comes from scattered operational sources; the needed data is extracted from the original data, processed, and integrated.
- Non-volatile: data isn’t updated; it exists mainly to serve decision analysis, and the operations involved are chiefly queries.
- Time-variant: stable data is kept in read-only form and doesn’t change over time.
- Used to support management decision making.
How Does a Data Warehouse Differ from a Database?
The databases we use every day (Oracle, MySQL, PostgreSQL) serve transaction processing — bank transfers, placing orders — and need very high responsiveness. They consist of tables, each describing a type of record: a user table describes a user’s name and password.
A data warehouse (Amazon Redshift, Hive) serves data analysis. It stores metadata, and while it also has tables, those tables may not describe things but rather a subject. A user table, for instance, holds more than a name and password: shaped by the dimensions needed for analysis, it can become very wide, containing all information about a user.
RenFei’s note: a database records a thing or an event and handles what’s happening right now; a data warehouse is built for analysis, classifying data by subject so it can be pulled and analyzed by category.
How Subjects Are Defined
As noted, a warehouse stores data by subject. A subject is an abstract concept for synthesizing, classifying and analyzing data, and one subject roughly corresponds to one macro analytical domain. Since this is a beginner course aimed at getting you moving fast, we won’t dig into deeper model design yet — just get to know the idea of a subject.
In practice subject domains get carved up differently depending on the perspective: by business, by requirement, by function or by application. There’s no fixed standard blueprint, and building one can be iterative — don’t obsess over abstracting every subject at once. Start with clearly defined subjects, then gradually generalize them into a standard model for your own industry.