Tutorial index: Big Data for Beginners: tutorial series
The previous big chapter covered the concept of a big data warehouse and how to think about building one. Now we start turning those ideas into reality. The foundation carrying all of it is the collection of big data components in the Hadoop ecosystem, which gradually becomes our data warehouse and platform.
What Is Hadoop?
Hadoop is a distributed system infrastructure developed by the Apache Foundation. Note the word infrastructure — it’s the bottom layer, so it can’t accomplish much on its own. It provides the ability to store and analyze massive datasets. Broadly speaking, “Hadoop” refers to the whole Hadoop ecosystem.
Three Major Distributions
Apache Hadoop
The Apache version is the original (and most basic) one, and the best starting point for learning.
Official site: http://hadoop.apache.org
Downloads: https://archive.apache.org/dist/hadoop/common/
Cloudera Hadoop
Founded in 2008, Cloudera was the first company to commercialize Hadoop, offering commercial Hadoop solutions to partners. In 2009 Hadoop’s creator, Doug Cutting, joined Cloudera. Cloudera’s product is primarily CDH.
Official site: https://www.cloudera.com/downloads/cdh.html
Downloads: https://docs.cloudera.com/documentation/enterprise/6/release-notes/topics/rg_cdh_6_download.html
Hortonworks
Founded in 2011, Hortonworks was a joint venture between Yahoo and Silicon Valley VC firm Benchmark Capital. At inception it took in roughly 25 to 30 Yahoo engineers dedicated to Hadoop — engineers who had helped Yahoo develop Hadoop since 2005 and had contributed about 80% of Hadoop’s code. Its flagship product was Hortonworks Data Platform (HDP).
Hadoop Components
Hadoop 1.x and 2.x differ somewhat. Hadoop 1.x consists of Common (supporting utilities), HDFS (data storage), and MapReduce (computation + scheduling).
Hadoop 2.x consists of Common (supporting utilities), HDFS (data storage), Yarn (resource scheduling), and MapReduce (computation).
Compared with 1.x, Hadoop 2.x adds Yarn for resource scheduling, leaving MapReduce concerned only with computation.
HDFS in Brief
The Hadoop Distributed File System (HDFS) is a critically important core capability. Highly fault-tolerant, it stores files split into blocks across distributed nodes with redundancy, so even a catastrophic failure of a single node doesn’t lose data.
- NameNode (nn): stores file metadata — filenames, directory structure, file attributes, plus each file’s block list and which DataNode holds each block.
- DataNode (dn): stores block data in the local filesystem, along with block checksums.
- Secondary NameNode (2nn): a helper background program monitoring HDFS state, taking snapshots of HDFS metadata at intervals.
Think of the NameNode as the index to the data, DataNode as the data itself, and the Secondary NameNode as a snapshot of the index. Since this is just a brief introduction to Hadoop, we’ll cover it properly later — no more detail here.
YARN in Brief
We mentioned Hadoop 2.x added Yarn for resource scheduling — but scheduling what? In computing, resources means CPU and memory, both of which are finite and therefore need allocating to whichever processes need them most.
ResourceManager (RM) is the resource manager. Job requests from external clients arrive at ResourceManager first; it represents all the cluster’s resources, monitors NodeManagers, and starts or monitors ApplicationMasters.
NodeManager (NM) manages the resources of a single node, handling commands from the ResourceManager and from ApplicationMasters.
ApplicationMaster (AM) handles data splitting, requesting resources for the application, assigning internal tasks, and monitoring and fault tolerance. When a task is submitted to ResourceManager, it picks a node and starts an ApplicationMaster to own that task — in effect, one person accountable per task. So every submitted job gets a corresponding ApplicationMaster following up on its execution and scheduling.
Container is an abstraction wrapping resources — memory, CPU, disk, network — and NodeManager schedules resources by opening and closing Containers.
MapReduce in Brief
MapReduce is really two parts, Map and Reduce — you might call it an idea and a processing algorithm.
Map splits the data into many pieces. Say we have 100 TB of data and want to find one record in it: Map first splits those 100 TB into a thousand pieces of 100 GB each and distributes them to the nodes, turning 100 TB into chunks of 100 GB, so a thousand nodes search in parallel and efficiency goes up.
Reduce collects the results from those thousand nodes and merges everyone’s answers into the final result.
This is a straight-to-the-point overview; we’ll go into detail later, so no more here.