Hadoop for Beginners (12): Hadoop's Writable Classes

Tutorial index: Big Data for Beginners: tutorial series

Last session we went through using MapReduce from code, where we met Writable classes for the first time. This section covers roughly what they’re for.

Why Serialization Is Needed

Hadoop is a distributed framework, which means data has to move between nodes or land on disk. Objects in memory must be serialized and then deserialized, and there is a great deal of that going on — hence Hadoop needs an efficient serialization mechanism, and that’s the Writable classes.

Java has Serializable for object serialization, but serializing with it drags along a lot of extra baggage: various validation data, inheritance information, and so on. Storing and shipping all that over network and disk I/O is wasteful and unnecessary, which is why Hadoop rolled its own Writable classes to carry objects.

Writable Classes in Hadoop

Last time we used LongWritable, Text, and IntWritable. There’s also BooleanWritable, ByteWritable, FloatWritable, and DoubleWritable. Here’s a quick mapping:

Java primitive	Writable implementation	Bytes
boolean	BooleanWritable	1
byte	ByteWritable	1
int	IntWritable	4
float	FloatWritable	4
long	LongWritable	8
double	DoubleWritable	8

Writing Your Own Writable Class

Hadoop only provides a handful of basic types, so if you need more you define your own object implementing Writable. That’s easy: implement org.apache.hadoop.io.Writable with its write and readFields methods, like this:

package net.renfei.hadoop.entity;
import org.apache.hadoop.io.Writable;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
/**
 * Title: DemoEntity
 * Description:
 *
 * @author RenFei
 */
public class DemoEntity implements Writable {
    private String ip;
    private String path;
    private int port;
    /**
     * Serialization
     *
     * @param dataOutput the data sink the framework gives us
     * @throws IOException
     */
    public void write(DataOutput dataOutput) throws IOException {
        dataOutput.writeUTF(ip);
        dataOutput.writeUTF(path);
        dataOutput.writeInt(port);
    }
    /**
     * Deserialization
     *
     * @param dataInput the data source the framework gives us
     * @throws IOException
     */
    public void readFields(DataInput dataInput) throws IOException {
        ip = dataInput.readUTF();
        path = dataInput.readUTF();
        port = dataInput.readInt();
    }
    public String getIp() {
        return ip;
    }
    public void setIp(String ip) {
        this.ip = ip;
    }
    public String getPath() {
        return path;
    }
    public void setPath(String path) {
        this.path = path;
    }
    public int getPort() {
        return port;
    }
    public void setPort(int port) {
        this.port = port;
    }
}

One thing to watch: the field order in write must match the order in readFields.

With that done, you can use your own DemoEntity for Hadoop input and output data.