13.1. Messaging Platform: Apache Kafka
Kafka is a case study in how a single abstraction — the distributed, replicated commit log — becomes the backbone of an entire data ecosystem. Its performance comes from working with the hardware rather than against it: sequential disk writes, zero-copy reads, and the page cache instead of a JVM heap. This chapter dissects the design decisions behind that performance, the KRaft mode that removed its ZooKeeper dependency, consumer-group rebalancing failure modes, and how exactly-once semantics are actually achieved.
Topics Covered
Section titled “Topics Covered”- 13.1.1. Disk-Based Persistence and Zero-Copy Optimization: Explains how Kafka’s sequential disk writes, page cache, and zero-copy reads achieve high throughput.
- 13.1.2. KRaft Mode: Eliminating the ZooKeeper Dependency: Covers KRaft mode, which removed Kafka’s ZooKeeper dependency by moving metadata into a Raft-based log.
- 13.1.3. Consumer Group Rebalancing: Failure Scenarios: Explains consumer-group rebalancing and the failure scenarios that cause stop-the-world pauses.
- 13.1.4. Exactly-Once Semantics: Idempotent Producer + Transactional API: Covers how Kafka achieves exactly-once with the idempotent producer and transactional API.
- 13.1.5. Kafka Streams vs. ksqlDB: When to Use Which: Compares Kafka Streams and ksqlDB for stream processing directly on top of Kafka.