4.1. Batch Processing
Batch processing computes over a finite, complete dataset — you have all the input before you start, which makes correctness easy and latency high. The defining challenge is not the map or reduce but the shuffle between them: moving terabytes across the network to co-locate related keys, where stragglers and skew decide your job’s runtime. This chapter traces the paradigm from MapReduce’s disk-bound model to Spark’s in-memory DAG, and the fault-tolerance and speculative-execution mechanics that keep a thousand-node job finishing.
Topics Covered
Section titled “Topics Covered”- 4.1.1. The MapReduce Paradigm: Functional Roots, Map, Shuffle, Reduce: Covers the MapReduce paradigm, its functional roots, and the map, shuffle, and reduce phases.
- 4.1.2. Distributed Sort: The Challenge of the Shuffle Phase: Explains why the shuffle phase, moving data across the network to sort it, dominates a batch job’s cost.
- 4.1.3. From Hadoop to Apache Spark: Disk vs. In-Memory Processing: Traces the shift from Hadoop’s disk-bound model to Spark’s in-memory DAG execution and its speedups.
- 4.1.4. RDD, DataFrame, and Dataset APIs: Compares Spark’s RDD, DataFrame, and Dataset APIs and the optimization each level enables.
- 4.1.5. Checkpointing and Fault Tolerance: Covers lineage and checkpointing, the mechanisms that let a failed task recompute without restarting the job.
- 4.1.6. Straggler Tasks: Speculative Execution: Explains how straggler tasks stall a job and how speculative execution races a backup copy to finish.