Health checks and failover

What you'll be able to do

1. 1Explain what liveness, local and dependency checks (and Kubernetes readiness) each prove, and why a check that tests a shared dependency can take every healthy server out of service at once.
2. 2Predict how long a failed server keeps receiving traffic and how many requests fail, from the probe interval, the failure threshold and the load.
3. 3Choose how checks react — shallow checks for the load balancer, deeper checks to a central monitor, fail open when every server looks sick — and how a database primary fails over to a standby.

Inside the lesson

steps

12

questions

13

simulations

4

You guess first, then run the system and see if you were right.

Step 1 · The setup

More on the Resilience Line

1. Failure modes and availabilityOpen9 min
2. Timeouts — capping the waitMembers9 min
3. Retries — with backoff and jitterMembers12 min
4. Circuit breakerMembers10 min
5. Bulkhead — walls between workloadsMembers12 min
6. Load shedding — turning the excess awayMembers12 min
7. Backpressure — a queue that says noMembers11 min
8. Rate limiting — a budget for each callerMembers12 min