Tutorial index: Big Data for Beginners: tutorial series
Earlier posts introduced HDFS. Now let’s dip into the other core Hadoop component: MapReduce.
What Is MapReduce?
MapReduce is a programming framework for distributed computing. Map and Reduce are its central ideas, and it originated at Google as a parallel computing model and method for large-scale data processing. Google designed MapReduce primarily to parallelize the processing of its enormous corpus of web pages for its search engine, and after inventing it, first used it to rewrite the Web document indexing system behind that engine. Since MapReduce applies broadly to many large-scale data problems, Google went on to use it widely across the company — tens of thousands of distinct algorithmic problems and programs there are handled with MapReduce.
In 2003 and 2004, Google published two conference papers on its distributed file system and MapReduce respectively, revealing the basic principles and design thinking behind GFS and MapReduce.
Strengths of MapReduce
- As a programming framework for distributed computing, it lets developers write distributed programs almost as simply as serial ones.
- When computing power falls short, you scale by simply adding machines.
- It’s fault-tolerant: when a node dies, its task moves to another node and the overall job doesn’t fail — none of that requires human intervention, the framework handles it automatically.
- Being a distributed computing framework, it can handle massive volumes of data.
Weaknesses of MapReduce
- It isn’t real-time. MapReduce can’t return answers instantly the way SQL can, so it’s mostly used for offline computation.
- It isn’t streaming. MapReduce was designed for static data, not for computing over live, dynamic data.
The MapReduce Idea
MapReduce splits into two stages — Map and Reduce. Let’s take them in turn.
Map
This stage is called “mapping” because the data we receive comes in all shapes and formats that may not suit our computation, so the raw data needs processing into the format we need. Consider the earlier WordCount word-counting example:
The raw data consists of lines of text, each holding several space-separated words, so Map reads the raw data line by line and splits it on spaces into K/V pairs, like [{renfei:1},{word:1},{renfei:1},{test:1}]. That completes the mapping.
Since the data is spread across many nodes as blocks, several nodes can execute in parallel at the same time, all working on it together.
Reduce
Reduce is the second stage, further processing the Map output. The counting itself in WordCount actually happens during Reduce: Map turned the source data into a format we recognize and can work with, so now we can process it — counting being one form of processing.
Wrapping Up
We now have a general sense of MapReduce. Next chapter we’ll implement the WordCount case from earlier in code, to deepen our understanding of how it works.