IV. Data Processing Architectures
Processing data at scale forces a choice between two shapes: bounded batches computed for throughput, and unbounded streams computed for latency. The industry spent a decade discovering that these are not two systems but two ends of one continuum, unified by the log. This part covers the batch paradigm (MapReduce to Spark), the stream paradigm and its hard problem of time (event time, watermarks, windowing), and the hybrid architectures that reconcile them.
Topics Covered
Section titled “Topics Covered”- 4.1. Batch Processing: Computing over finite datasets: MapReduce to Spark, the shuffle bottleneck, fault tolerance, and speculative execution.
- 4.2. Stream Processing: Computing over unbounded data: event time versus processing time, windowing, watermarks, triggers, and Apache Flink.
- 4.3. Hybrid Architectures and the Log Abstraction: The commit log as the source of truth: Lambda and Kappa architectures, log compaction, and Change Data Capture.