Hadoop for Beginners (7): HDFS Data Read and Write Flow
Last time we could already operate HDFS from code, but what actually happens inside the Hadoop cluster? This post gives a quick overview of how HDFS reads and writes data.
TagAll posts
#Development
219 posts
More Posts
Page 8 of 22Tutorial: Building a Custom Apache NiFi Processor
This tutorial walks you through creating a custom Apache NiFi Processor project and writing your own processor to handle FlowFiles. Source code: https://github.com/renfei/demo/tree/master/nifi/nifi-custom-bundle
Hadoop for Beginners (6): Using the Hadoop API to Drive HDFS from Code
Last post covered operating HDFS with shell commands, but in practice we can't keep doing everything by hand — we need automation through code. This post gets you familiar with controlling files in HDFS from Java.
Verifying getInputStream Works for Files Inside a Spring Boot Jar
I previously wrote "The Big Pitfall of Reading Files After Packaging a Spring Boot Project Into a Jar: ClassPathResource Fails to Read Files on the Classpath" and published it on my blog and CSDN. I hadn't logged into CSDN in a long while; today I did, and found two commenters both saying they tried my approach and couldn't get anything.
Hadoop for Beginners (5): HDFS, the Distributed File System
HDFS stands for Hadoop Distributed File System. It is a highly fault-tolerant system designed for deployment on cheap commodity machines. It delivers high-throughput data access, which makes it a great fit for applications over very large data sets.
Hadoop for Beginners (4): A Fully Distributed Hadoop Cluster
Last time we tried installing a pseudo-distributed setup, but what you actually use in production is a distributed cluster — that's the real point. This post builds a minimal Hadoop cluster on three nodes to experience a fully distributed environment.
Hadoop for Beginners (3): Single-Node Local and Pseudo-Distributed Mode
Since this is a beginner course and many people's machines can't handle a real cluster, let's start with Hadoop's single-node mode so you can run simple operations with Hadoop MapReduce and HDFS. This section is basically a Hello World for Hadoop — pulling the mystery off it.
Hadoop for Beginners (2): Installing Hadoop
Before we can start our Hadoop journey we need to know how to install it. Later we'll experiment and learn across multiple Hadoop nodes; this post walks through the installation, which is quite simple.
Updating discuz-ucenter-api-for-java for HTTPS (SSL) Sites
As the web has evolved, more sites are served over HTTPS. The discuz-ucenter-api-for-java library that connects Discuz's UCenter to Java runs into trouble over HTTPS: it's a very old project that hasn't been updated in ages and doesn't support HTTPS endpoints. So I updated it.
Connecting DBeaver to a DaMeng Database
Newer DBeaver versions no longer support ODBC connections, probably because the JDBC-ODBC bridge was removed from Java 8. So the old tutorials telling you to use ODBC no longer work. Here's the correct way to connect DBeaver to DaMeng.
