3.1. Data Partitioning / Sharding
Partitioning (sharding) splits a dataset across nodes so that no single machine holds — or is bottlenecked by — all of it. The partitioning scheme you choose determines everything downstream: range partitioning enables efficient scans but invites hot spots, hash partitioning spreads load evenly but destroys locality, and any scheme must survive nodes joining and leaving without a full reshuffle. This chapter covers those trade-offs, consistent hashing as the standard answer to rebalancing, and the secondary-index problem that partitioning quietly creates.
Topics Covered
Section titled “Topics Covered”- 3.1.1. Key-Range Partitioning: Advantages and Range Scans: Covers range partitioning, which keeps keys ordered for efficient scans at the cost of hot-spot risk.
- 3.1.2. Hash Partitioning: Uniform Distribution: Explains hash partitioning, which spreads load uniformly but sacrifices range-scan locality.
- 3.1.3. Skew and Hot-Spot Problems: Real-World Scenarios: Analyzes real-world skew and hot-spot scenarios and the salting techniques that mitigate them.
- 3.1.4. Consistent Hashing and Virtual Nodes (vnodes): Introduces consistent hashing and virtual nodes as the standard answer to rebalancing without a full reshuffle.
- 3.1.5. Dynamic Rebalancing: Zero-Downtime Strategies: Covers zero-downtime strategies for moving partitions as nodes join and leave the cluster.
- 3.1.6. Colocated Joins: Placing Related Data on the Same Shard: Explains placing related data on the same shard so joins stay local instead of crossing the network.
- 3.1.7. Secondary Indexes in Partitioned Data: Local vs. Global: Contrasts local (document-partitioned) and global (term-partitioned) secondary indexes and their read/write costs.