Hadoop for Beginners (2): Installing Hadoop
Before we can start our Hadoop journey we need to know how to install it. Later we'll experiment and learn across multiple Hadoop nodes; this post walks through the installation, which is quite simple.
TagAll posts
#Big Data
34 posts
More Posts
Page 3 of 4Connecting DBeaver to a DaMeng Database
Newer DBeaver versions no longer support ODBC connections, probably because the JDBC-ODBC bridge was removed from Java 8. So the old tutorials telling you to use ODBC no longer work. Here's the correct way to connect DBeaver to DaMeng.
Correction to "Data Extraction Methods in Big Data ETL"
The original post derived two complements, CuB and CuA, from the first comparison, then pulled the real data for a second comparison. Applying this in production revealed that pulling the source data in the second comparison can be skipped, so I've corrected the ETL steps from that post.
Hadoop for Beginners (1): What Hadoop Is and What It's Made Of
The previous big chapter covered the concept of a big data warehouse and how to think about building one. Now we start turning those ideas into reality. The foundation carrying all of it is the collection of big data components in the Hadoop ecosystem, which gradually becomes our data warehouse and platform.
Locality-Sensitive Hashing for Faster Big Data ETL Comparison
Last post hashed the digest of every field in each row, compressing dozens of fields into a single hash to speed up comparison. But that only compressed the fields — the row count stayed the same. With billions of rows we still can't find the changed records fast enough.
Data Warehouse Concepts (4): Full, Incremental, Zipper, Transaction and Snapshot Tables
Last post covered fact tables, dimension tables, the star schema and the snowflake schema. Beyond those there's more industry jargon to learn. This post explains the big data slang: full tables, incremental tables, zipper tables, transaction tables, and snapshot tables. Some of it may not click yet — it will once you work with Hive — so take this as foundational knowledge.
Data Warehouse Concepts (3): Data Modeling — Fact Tables, Dimension Tables, Star and Snowflake Schemas
In the previous post on data warehouse layering we mentioned that each layer does some processing beyond the raw tables, and that processing involves fact tables and dimension tables. This post gives you a rough understanding of fact tables, dimension tables, and the star and snowflake data models.
Data Warehouse Concepts (2): Layered Data Warehouse Design
As data flows into our warehouse over time, both the variety and volume grow. Without governance and design, querying and retrieving that data becomes very painful — so the warehouse needs designing, with each type of data filed where it belongs, ready to be called up whenever we need it.
Data Warehouse Concepts (1): What a Data Warehouse Is and How It Differs from a Database
This is the opening post of my big data beginner series. Every big data component and tool we use exists to solve data problems, and data has to be stored and retrieved through a data warehouse — no technical architecture escapes that. So let's start by understanding what a data warehouse is, and how it differs from a database.
Big Data for Beginners: Tutorial Series
Covers data warehouse concepts; beginner tutorials for Hadoop, ZooKeeper, Hive, Flume, Kafka, Hbase, Sqoop, Oozie, Azkaban, Kylin, CDH, Impala, Hue, ClickHouse, Kettle, Ambari, ELK, Scala and Flink.
