7.1. Flow Control and Overload Management
Every system has a finite capacity, and the difference between graceful degradation and total collapse is what happens when demand exceeds it. Without flow control, an overloaded system does not slow down — it falls into a metastable failure where retries and queues amplify the load until nothing gets through, even after the original trigger is gone. This chapter starts from queueing theory (why latency explodes near saturation), then covers backpressure, load shedding, rate limiting, and the retry-storm dynamics that turn a blip into an outage.
Topics Covered
Section titled “Topics Covered”- 7.1.1. Queueing Theory Foundations: Little’s Law and the Utilization Curve: Grounds flow control in queueing theory: Little’s Law and why latency explodes as utilization nears saturation.
- 7.1.2. Backpressure: Pushing Load Back to the Source: Covers backpressure, propagating overload signals upstream so producers slow down instead of overwhelming consumers.
- 7.1.3. Load Shedding: Which Requests to Drop and How: Explains load shedding: deliberately dropping requests to keep the system alive when demand exceeds capacity.
- 7.1.4. Priority Queues: Protecting Critical Requests: Covers priority queues that protect critical requests by serving them ahead of best-effort traffic.
- 7.1.5. Rate Limiting: Token Bucket, Leaky Bucket, Fixed and Sliding Window: Covers the rate-limiting algorithms: token bucket, leaky bucket, and fixed and sliding window counters.
- 7.1.6. Adaptive Rate Limiting: Dynamic Limits: Explains adaptive rate limiting that adjusts limits dynamically based on real-time system health.
- 7.1.7. Retry Storms and Metastable Failures: Work Amplification and Retry Budgets: Covers retry storms, work amplification, and the retry budgets that prevent metastable failure.