167 posts
The previous tutorial introduced how MapReduce executes and sketched out InputFormat. Now it's time to talk about Shuffle. The data processing that happens after the Map method and before the Reduce method is called Shuffle.
In the previous post we got hands-on with MapReduce by writing some code. The MapReduce workflow looks roughly like this.
Last session we went through using MapReduce from code, where we met Writable classes for the first time. This section covers roughly what they're for.
Last post gave us a general idea of what MapReduce is. In this one we'll implement WordCount in code and get a feel for how MapReduce actually works.
Earlier posts introduced HDFS. Now let's dip into the other core Hadoop component: MapReduce.
On March 18, 2021, GitLab — the world's second-largest open source software company — announced that JiHu Information Technology (Hubei) Co., Ltd. has officially registered in Wuhan, China.
Last post covered HDFS broadly. From this section on we'll drive Hadoop by writing code, so first we need a local Hadoop development environment. If you followed the earlier posts and got Hadoop running in a VM, setting it up locally is a walk in the park. If you haven't — or haven't read those posts — read them first.
When a DataNode starts it registers itself with the NameNode and periodically (every hour) reports all of its block information. It also exchanges heartbeats with the NameNode every 3 seconds, and those heartbeats carry commands for the DataNode. If no heartbeat arrives for over 10 minutes, the node is considered unavailable.
Last time we could already operate HDFS from code, but what actually happens inside the Hadoop cluster? This post gives a quick overview of how HDFS reads and writes data.
This tutorial walks you through creating a custom Apache NiFi Processor project and writing your own processor to handle FlowFiles. Source code: https://github.com/renfei/demo/tree/master/nifi/nifi-custom-bundle
