Hadoop for Beginners (8): How DataNode Works

Tutorial index: Big Data for Beginners: tutorial series

How DataNode Works

When a DataNode starts it registers itself with the NameNode and periodically (every hour) reports all of its block information.

It also exchanges heartbeats with the NameNode every 3 seconds, and those heartbeats carry commands for the DataNode. If no heartbeat arrives for over 10 minutes, the node is considered unavailable.

Data Integrity Checks

To ensure data is transmitted correctly, integrity checks are performed using the CRC algorithm: a checksum is computed over the data, and if it doesn’t match, the block is considered corrupt. The Client reports the bad block and the DataNode holding it to the NameNode.

The NameNode marks the block “corrupt” — once marked, it won’t point clients at that block, and won’t use it to replicate to other DataNodes.

The NameNode copies a good replica of the block to another DataNode, then deletes the bad one.

DataNode Timeout Settings

You can configure the timeout for declaring a DataNode dead via heartbeat.recheck.interval in hdfs-site.xml (in milliseconds) and dfs.heartbeat.interval (in seconds).

Adding DataNodes

Hadoop scales well, so adding a DataNode is very simple: following the earlier tutorials, configure a DataNode and just start it — it registers with the NameNode and joins the cluster.

If the new machine leaves data unbalanced, run sbin/start-balancer.sh to redistribute blocks evenly.

Removing DataNodes

To evict a DataNode from the cluster you can use allowlists or denylists, but the same hostname must never appear in both at once.

Allowlist

Hosts on the allowlist may access the NameNode; hosts not on it are made to leave.

Create a dfs.hosts file under /opt/module/hadoop-2.10.1/etc/hadoop on the NameNode listing the hostnames of every node in the cluster. Adjust the path for your own setup.

Add the dfs.hosts property to the NameNode’s hdfs-site.xml:

<property>
    <name>dfs.hosts</name>
    <value>/opt/module/hadoop-2.10.1/etc/hadoop/dfs.hosts</value>
</property>

Distribute the config file to every node, then refresh the NameNode and update ResourceManager nodes:

hdfs dfsadmin -refreshNodes
yarn rmadmin -refreshNodes

Denylist

Hosts on the denylist are forced out.

Create a dfs.hosts.exclude file under /opt/module/hadoop-2.10.1/etc/hadoop on the NameNode with the hostnames to evict. Again, use your own path.

Add the dfs.hosts.exclude property to the NameNode’s hdfs-site.xml:

<property>
    <name>dfs.hosts.exclude</name>
    <value>/opt/module/hadoop-2.10.1/etc/hadoop/dfs.hosts.exclude</value>
</property>

Refresh the NameNode and ResourceManager:

hdfs dfsadmin -refreshNodes
yarn rmadmin -refreshNodes

Wait until the decommissioning node reaches the decommissioned state (meaning all its blocks have finished replicating), then stop that node and its node manager. Note: with a replication factor of 3 you can’t successfully decommission while three or fewer nodes are in service — change the replication factor first.