Tutorial index: Big Data for Beginners: tutorial series
Last post covered operating HDFS with shell commands, but in practice we can’t keep doing everything by hand — we need automation through code. This post gets you familiar with controlling files in HDFS from Java.
Prerequisites
This chapter assumes you can already write Java — Java SE fundamentals and Maven builds included. If you aren’t comfortable building projects with Java/Maven yet, learn that first. It also assumes you’ve read the earlier chapters and stood up a Hadoop platform; without one you won’t be able to follow along. Complete source for this chapter: https://github.com/renfei/demo/tree/master/hadoop/hadoop_api
Creating the Maven Project
Start by creating a Maven project and pulling in the dependencies. What Maven is, is out of scope — we’re here for HDFS, so learn Maven on your own time. Here are the dependencies from my pom.xml:
<properties>
<hadoop.version>2.10.1</hadoop.version>
<log4j.version>2.14.0</log4j.version>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>RELEASE</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j.version}</version>
</dependency>
</dependencies>HDFS API Operations
Creating a Directory
public void mkdirs() throws IOException, InterruptedException {
FileSystem fileSystem = FileSystem.get(uri, configuration, user);
fileSystem.mkdirs(new Path("/demo"));
fileSystem.close();
}Full source: https://github.com/renfei/demo/blob/master/hadoop/hadoop_api/src/main/java/net/renfei/hadoop/HDFSClient.java#L38
Uploading a File
public void upload() throws IOException, InterruptedException {
FileSystem fileSystem = FileSystem.get(uri, configuration, user);
fileSystem.copyFromLocalFile(new Path("/Users/renfei/Downloads/demo.txt"), new Path("/demo/demo.txt"));
fileSystem.close();
}Full source: https://github.com/renfei/demo/blob/master/hadoop/hadoop_api/src/main/java/net/renfei/hadoop/HDFSClient.java#L51
Downloading a File
public void get() throws IOException, InterruptedException {
FileSystem fileSystem = FileSystem.get(uri, configuration, user);
fileSystem.copyToLocalFile(new Path("/demo/demo.txt"), new Path("/Users/renfei/Downloads/demo2.txt"));
fileSystem.close();
}Full source: https://github.com/renfei/demo/blob/master/hadoop/hadoop_api/src/main/java/net/renfei/hadoop/HDFSClient.java#L64
Renaming a File
public void rename() throws IOException, InterruptedException {
FileSystem fileSystem = FileSystem.get(uri, configuration, user);
fileSystem.rename(new Path("/demo/demo.txt"), new Path("/demo/demo2.txt"));
fileSystem.close();
}Full source: https://github.com/renfei/demo/blob/master/hadoop/hadoop_api/src/main/java/net/renfei/hadoop/HDFSClient.java#L77
Getting File Details
public void listFiles() throws IOException, InterruptedException {
FileSystem fileSystem = FileSystem.get(uri, configuration, user);
RemoteIterator<LocatedFileStatus> listFiles = fileSystem.listFiles(new Path("/demo/"), true);
while (listFiles.hasNext()) {
LocatedFileStatus status = listFiles.next();
// File name
System.out.println(status.getPath().getName());
// Length
System.out.println(status.getLen());
// Permissions
System.out.println(status.getPermission());
// Group
System.out.println(status.getGroup());
// Stored block info
BlockLocation[] blockLocations = status.getBlockLocations();
for (BlockLocation blockLocation : blockLocations) {
// Host nodes storing this block
String[] hosts = blockLocation.getHosts();
for (String host : hosts) {
System.out.println(host);
}
}
System.out.println("---------------------");
}
fileSystem.close();
}Full source: https://github.com/renfei/demo/blob/master/hadoop/hadoop_api/src/main/java/net/renfei/hadoop/HDFSClient.java#L91
Deleting a File
public void delete() throws IOException, InterruptedException {
FileSystem fileSystem = FileSystem.get(uri, configuration, user);
// The second argument, true, means recursive deletion
if (fileSystem.delete(new Path("/demo"), true)) {
System.out.println("deleted successfully");
} else {
System.out.println("delete failed");
}
fileSystem.close();
}Full source: https://github.com/renfei/demo/blob/master/hadoop/hadoop_api/src/main/java/net/renfei/hadoop/HDFSClient.java#L126
Streaming Upload
public void upload() throws IOException, InterruptedException {
FileSystem fileSystem = FileSystem.get(uri, configuration, user);
// Create a file input stream
FileInputStream fileInputStream = new FileInputStream(new File("/Users/renfei/Downloads/demo.txt"));
// Get the output stream
FSDataOutputStream fsDataOutputStream = fileSystem.create(new Path("/demo/demo.txt"));
// Copy between streams
IOUtils.copyBytes(fileInputStream, fsDataOutputStream, configuration);
// Close resources
IOUtils.closeStream(fsDataOutputStream);
IOUtils.closeStream(fileInputStream);
fileSystem.close();
}Streaming Download
public void get() throws IOException, InterruptedException {
FileSystem fileSystem = FileSystem.get(uri, configuration, user);
// Get the input stream
FSDataInputStream fsDataOutputStream = fileSystem.open(new Path("/demo/demo.txt"));
// Get the output stream
FileOutputStream fileOutputStream = new FileOutputStream(new File("/Users/renfei/Downloads/demo2.txt"));
// Copy between streams
IOUtils.copyBytes(fsDataOutputStream, fileOutputStream, configuration);
// Close resources
IOUtils.closeStream(fsDataOutputStream);
IOUtils.closeStream(fileOutputStream);
fileSystem.close();
}