7.3. Failure Detection and Membership Protocols
Before you can react to a failure, you have to detect it — and in an asynchronous network you cannot distinguish a crashed node from a slow one. Every failure detector is a bet on that ambiguity: too aggressive and you evict healthy nodes under load, too patient and you serve requests to the dead. This chapter covers heartbeats and the timeout science behind them, gossip protocols that scale membership without a central monitor, the phi-accrual detector that treats failure as a probability, and the liveness-versus-readiness distinction that trips up every Kubernetes operator.
Topics Covered
Section titled “Topics Covered”- 7.3.1. Heartbeats and the Science of Timeout Configuration: Covers heartbeats and the timeout science behind them: the trade-off between fast detection and false positives.
- 7.3.2. Gossip Protocols: Information Dissemination via Epidemic Algorithms: Explains gossip protocols that disseminate membership information epidemically without a central monitor.
- 7.3.3. Phi Accrual Failure Detector: Viewing Failure as a Probability: Covers the phi-accrual detector that outputs a continuous suspicion level instead of a binary alive/dead verdict.
- 7.3.4. SWIM Protocol: Scalable Membership: Explains the SWIM protocol for scalable membership with separated failure detection and dissemination.
- 7.3.5. Liveness vs. Readiness: The Difference Between Kubernetes Probes: Clarifies the liveness-versus-readiness distinction and why confusing Kubernetes probes causes restart storms.