167 posts
Last post covered operating HDFS with shell commands, but in practice we can't keep doing everything by hand — we need automation through code. This post gets you familiar with controlling files in HDFS from Java.
I previously wrote "The Big Pitfall of Reading Files After Packaging a Spring Boot Project Into a Jar: ClassPathResource Fails to Read Files on the Classpath" and published it on my blog and CSDN. I hadn't logged into CSDN in a long while; today I did, and found two commenters both saying they tried my approach and couldn't get anything.
HDFS stands for Hadoop Distributed File System. It is a highly fault-tolerant system designed for deployment on cheap commodity machines. It delivers high-throughput data access, which makes it a great fit for applications over very large data sets.
Last time we tried installing a pseudo-distributed setup, but what you actually use in production is a distributed cluster — that's the real point. This post builds a minimal Hadoop cluster on three nodes to experience a fully distributed environment.
Since this is a beginner course and many people's machines can't handle a real cluster, let's start with Hadoop's single-node mode so you can run simple operations with Hadoop MapReduce and HDFS. This section is basically a Hello World for Hadoop — pulling the mystery off it.
Before we can start our Hadoop journey we need to know how to install it. Later we'll experiment and learn across multiple Hadoop nodes; this post walks through the installation, which is quite simple.
As the web has evolved, more sites are served over HTTPS. The discuz-ucenter-api-for-java library that connects Discuz's UCenter to Java runs into trouble over HTTPS: it's a very old project that hasn't been updated in ages and doesn't support HTTPS endpoints. So I updated it.
Newer DBeaver versions no longer support ODBC connections, probably because the JDBC-ODBC bridge was removed from Java 8. So the old tutorials telling you to use ODBC no longer work. Here's the correct way to connect DBeaver to DaMeng.
The original post derived two complements, CuB and CuA, from the first comparison, then pulled the real data for a second comparison. Applying this in production revealed that pulling the source data in the second comparison can be skipped, so I've corrected the ETL steps from that post.
MariaDB/MySQL is case-sensitive on Linux, so any mismatch in capitalization between your table or column names and your SQL throws errors. That's why we usually turn case sensitivity off first.
