Z3 · Resilience & Reliability

Load shedding

Silver

When a server nears its capacity it starts turning away the excess requests with a fast, cheap error, so the requests it does accept still finish quickly — instead of slowing down for everyone until most requests time out.

Use when: Any service that can get more traffic than it can serve — sudden surges, lost servers, a shift to costlier requests · Buying time while autoscaling adds servers
Avoid when: As a substitute for capacity — shedding all the time means the fleet is too small · When a rejection costs as much as serving the request — make the rejection path cheap first
Trade-offs: Keeps accepted requests fast, but the excess clients see errors · A threshold set too low rejects requests the service could have handled · Rejected requests need their own metrics, or they distort latency and hide the shedding
Requires: Timeouts, Server capacity & scaling up

3 objectives · 9 misconceptions

explain Explain why quickly rejecting the excess keeps more requests succeeding than trying to serve them all.
predict Predict goodput and latency for a server pushed past its capacity, with and without load shedding.
choose Choose where to shed (gateway, load balancer or server), when to start, and which requests to turn away first.

Not true: Serving every request, even slowly, is better than turning some away.
Not true: With autoscaling in place, a service never needs to shed load.
Not true: A server should start rejecting requests only once it is fully saturated.
Not true: Rejecting a request costs the server nothing.
Not true: Holding excess requests in a queue until the servers are free is safer than rejecting them.
Not true: When shedding, it doesn't matter which requests are turned away.
Not true: If median latency stays low while shedding, the service is healthy.
Not true: A server pushed past full keeps serving what it can and turns only the excess away.
Not true: A server held just under full keeps slowing down, as if it were past full.

Learn

Load shedding — turning the excess awayMembers~12 min

Practice

Sale dayMembersmedium

Check

Station test-out80% to pass

Neighborhood

builds onTimeouts Server capacity & scaling up

→Load shedding→

unlocksBackpressure Rate limiting

contrast withRate limiting

On linesResilience Line