Health checks and failover
Resilience & Reliability
Start lesson · ~15 minMembers
A health check asks each server whether it can do useful work, so traffic stops going to the broken ones. What it tests matters. A shallow check only proves the process answers; a deep check also tries the server's dependencies, and when a shared dependency fails every server fails its check at once — the fleet empties and a partial outage becomes a full one. Detection takes the probe interval times the failure threshold, so a dead server still gets traffic for a while; tuning it faster makes healthy servers flap. A stateful primary can't simply be routed around — a standby has to be promoted, which takes downtime and must avoid two primaries at once.
- Use when
- Every fleet behind a load balancer or orchestrator — a shallow check to route around dead or stuck servers · Readiness during startup and warm-up, so new servers get traffic only once they can serve it · A database or other stateful primary that must keep running when its machine or zone fails — a standby with automatic failover
- Avoid when
- A deep dependency check wired straight to the load balancer with nothing to stop it removing every server at once · Restart-on-failure (liveness) checks for problems a restart can't fix, such as a dependency that is down
- Trade-offs
- Deeper checks catch more local faults, but a shared-dependency failure trips every server together · Faster detection means fewer failed requests, but more false alarms and servers flapping in and out · A standby makes a database survive its machine, but writes pay for synchronous copying and failover still takes about a minute
3 objectives · 10 misconceptions
- Explain what liveness, local and dependency checks (and Kubernetes readiness) each prove, and why a check that tests a shared dependency can take every healthy server out of service at once.
- Predict how long a failed server keeps receiving traffic and how many requests fail, from the probe interval, the failure threshold and the load.
- Choose how checks react — shallow checks for the load balancer, deeper checks to a central monitor, fail open when every server looks sick — and how a database primary fails over to a standby.
- Not true: The deeper a health check tests the dependencies, the safer the service.
- Not true: A server whose process is up and listening on its port is healthy.
- Not true: The load balancer stops sending traffic to a server the moment it dies.
- Not true: Checking more often with a lower failure threshold is always better.
- Not true: When every server fails its health check, the load balancer should take them all out of service.
- Not true: Restarting a server fixes a health check that fails because its database is down.
- Not true: Under heavy load, a server should answer user requests before health checks.
- Not true: Failing over a database is just sending traffic to another copy, like taking a web server out of rotation.
- Not true: A database's failover standby can carry read traffic while it waits.
- Not true: If the old primary comes back after a failover, both copies can safely accept writes.