VII. Resilience & Production Engineering
In a large enough system, something is always failing — the question is never how to prevent failure but how to keep the system serving despite it. Resilience is the discipline of designing for that reality: shedding load before you collapse, isolating failures before they cascade, detecting dead nodes quickly without falsely accusing slow ones, and deliberately breaking things in production to find weaknesses first. This part covers flow control, the protection patterns (circuit breaker, bulkhead, retry), failure detection, and chaos engineering.
Topics Covered
Section titled “Topics Covered”- 7.1. Flow Control and Overload Management: Surviving overload: queueing theory, backpressure, load shedding, rate limiting, and retry-storm dynamics.
- 7.2. Protection Patterns: Isolating a failing dependency: circuit breaker, bulkhead, retry with backoff, timeouts, and fallback strategies.
- 7.3. Failure Detection and Membership Protocols: Distinguishing dead from slow: heartbeats, gossip protocols, the phi-accrual detector, and Kubernetes liveness versus readiness.
- 7.4. Chaos Engineering: Injecting failure on purpose: Chaos Monkey, the principles of chaos, fault injection, GameDay drills, and automation.