Hadoop for Beginners (11): Writing MapReduce in Code — a WordCount Example
Last post gave us a general idea of what MapReduce is. In this one we'll implement WordCount in code and get a feel for how MapReduce actually works.
TagAll posts
#Big Data
34 posts
More Posts
Page 2 of 4Hadoop for Beginners (10): First Look at MapReduce Distributed Computing
Earlier posts introduced HDFS. Now let's dip into the other core Hadoop component: MapReduce.
Hadoop for Beginners (9): Setting Up a Local Hadoop Development Environment
Last post covered HDFS broadly. From this section on we'll drive Hadoop by writing code, so first we need a local Hadoop development environment. If you followed the earlier posts and got Hadoop running in a VM, setting it up locally is a walk in the park. If you haven't — or haven't read those posts — read them first.
Hadoop for Beginners (8): How DataNode Works
When a DataNode starts it registers itself with the NameNode and periodically (every hour) reports all of its block information. It also exchanges heartbeats with the NameNode every 3 seconds, and those heartbeats carry commands for the DataNode. If no heartbeat arrives for over 10 minutes, the node is considered unavailable.
Hadoop for Beginners (7): HDFS Data Read and Write Flow
Last time we could already operate HDFS from code, but what actually happens inside the Hadoop cluster? This post gives a quick overview of how HDFS reads and writes data.
Tutorial: Building a Custom Apache NiFi Processor
This tutorial walks you through creating a custom Apache NiFi Processor project and writing your own processor to handle FlowFiles. Source code: https://github.com/renfei/demo/tree/master/nifi/nifi-custom-bundle
Hadoop for Beginners (6): Using the Hadoop API to Drive HDFS from Code
Last post covered operating HDFS with shell commands, but in practice we can't keep doing everything by hand — we need automation through code. This post gets you familiar with controlling files in HDFS from Java.
Hadoop for Beginners (5): HDFS, the Distributed File System
HDFS stands for Hadoop Distributed File System. It is a highly fault-tolerant system designed for deployment on cheap commodity machines. It delivers high-throughput data access, which makes it a great fit for applications over very large data sets.
Hadoop for Beginners (4): A Fully Distributed Hadoop Cluster
Last time we tried installing a pseudo-distributed setup, but what you actually use in production is a distributed cluster — that's the real point. This post builds a minimal Hadoop cluster on three nodes to experience a fully distributed environment.
Hadoop for Beginners (3): Single-Node Local and Pseudo-Distributed Mode
Since this is a beginner course and many people's machines can't handle a real cluster, let's start with Hadoop's single-node mode so you can run simple operations with Hadoop MapReduce and HDFS. This section is basically a Hello World for Hadoop — pulling the mystery off it.
