III. Data Management & Consistency

Data is the hardest thing to distribute, because splitting and copying it forces a choice the network will not let you avoid: when a partition happens, do you sacrifice consistency or availability? This part works through the full stack of that decision — how to partition data across nodes, how to replicate it for durability and reads, which consistency model to expose, what storage engine writes it to disk, and how caching trades freshness for speed. Every chapter here is an exercise in choosing which guarantee to give up.

Topics Covered

• 3.1. Data Partitioning / Sharding: Splitting datasets across nodes: range and hash partitioning, hot spots, consistent hashing, and partitioned secondary indexes.
• 3.2. Replication Strategies: Keeping data copies in sync: single-leader, multi-leader, and leaderless topologies, split-brain risk, replication lag, and anti-entropy.
• 3.3. Consistency Models: The spectrum of what a read may return: linearizability, serializability, causal and eventual consistency, CAP and PACELC.
• 3.4. Storage Engines and Data Structures: How databases write bytes to disk: LSM-trees versus B-trees, SSTables, Bloom filters, the write-ahead log, and columnar layouts.
• 3.5. Distributed Caching: Trading freshness for speed: cache strategies, invalidation, Redis cluster architecture, and stampede prevention.