547 posts
Last post covered HDFS broadly. From this section on we'll drive Hadoop by writing code, so first we need a local Hadoop development environment. If you followed the earlier posts and got Hadoop running in a VM, setting it up locally is a walk in the park. If you haven't — or haven't read those posts — read them first.
When a DataNode starts it registers itself with the NameNode and periodically (every hour) reports all of its block information. It also exchanges heartbeats with the NameNode every 3 seconds, and those heartbeats carry commands for the DataNode. If no heartbeat arrives for over 10 minutes, the node is considered unavailable.
Last time we could already operate HDFS from code, but what actually happens inside the Hadoop cluster? This post gives a quick overview of how HDFS reads and writes data.
This tutorial walks you through creating a custom Apache NiFi Processor project and writing your own processor to handle FlowFiles. Source code: https://github.com/renfei/demo/tree/master/nifi/nifi-custom-bundle
Last post covered operating HDFS with shell commands, but in practice we can't keep doing everything by hand — we need automation through code. This post gets you familiar with controlling files in HDFS from Java.
I previously wrote "The Big Pitfall of Reading Files After Packaging a Spring Boot Project Into a Jar: ClassPathResource Fails to Read Files on the Classpath" and published it on my blog and CSDN. I hadn't logged into CSDN in a long while; today I did, and found two commenters both saying they tried my approach and couldn't get anything.
HDFS stands for Hadoop Distributed File System. It is a highly fault-tolerant system designed for deployment on cheap commodity machines. It delivers high-throughput data access, which makes it a great fit for applications over very large data sets.
Last time we tried installing a pseudo-distributed setup, but what you actually use in production is a distributed cluster — that's the real point. This post builds a minimal Hadoop cluster on three nodes to experience a fully distributed environment.
Somewhere along my long history with macOS I hit a bizarre failure from unknown causes, and Apple support suggested I create a new account. So I changed my username from "neil" to "renfei". After copying my data over from the old home folder, various apps kept throwing errors — and judging by the messages, they were still looking for paths under the old username.
Since this is a beginner course and many people's machines can't handle a real cluster, let's start with Hadoop's single-node mode so you can run simple operations with Hadoop MapReduce and HDFS. This section is basically a Hello World for Hadoop — pulling the mystery off it.
