4.3. Hybrid Architectures and the Log Abstraction
The commit log is the abstraction that collapses the batch/stream divide: an append-only, replayable sequence of records that a batch job and a stream processor can both consume. Once you see the log as the source of truth, Lambda architecture (maintaining separate batch and speed layers) reveals itself as accidental complexity, and Kappa (everything is a stream, reprocess by replaying) becomes the simplification. This chapter covers that progression, log compaction, and Change Data Capture as the bridge from databases into the streaming world.
Topics Covered
Section titled “Topics Covered”- 4.3.1. The Commit Log Abstraction: The Core Idea Behind Kafka/Pulsar: Establishes the append-only commit log as the unifying abstraction behind Kafka and Pulsar.
- 4.3.2. Lambda Architecture: The Complexity of Batch + Speed Layers: Analyzes Lambda architecture and the accidental complexity of maintaining separate batch and speed layers.
- 4.3.3. Kappa Architecture: Everything Is a Stream: Explains Kappa architecture, which treats everything as a stream and reprocesses by replaying the log.
- 4.3.4. Log Compaction: Kafka Compacted Topics and Tombstones: Covers Kafka log compaction, tombstones, and the cleaner thread that retains only the latest value per key.
- 4.3.5. Change Data Capture (CDC): Real-Time Synchronization: Explains Change Data Capture, streaming a database’s changes in real time from its transaction log.
- 4.3.6. Debezium: Practical Database CDC Implementation: Covers Debezium’s practical CDC implementation, snapshot-then-stream coordination, and ordering guarantees.