Load leveling
Load leveling
Silver
A queue in front of a service with a fixed capacity — such as an outside provider that caps how fast you may send — lets that service work at its own steady rate. Peaks wait in the queue, so the service is sized for the average, not the peak. The price is delay — a peak leaves a backlog that drains only with the capacity left over from normal traffic.
- Use when
- A downstream with a limit you can't raise — an outside provider's send cap, a shared database · Peaks well above the average, for work that can wait a while · Keeping the producer working while the consumer is down for a deploy or an outage
- Avoid when
- The caller needs the result now · Load that is low and steady — the queue adds parts for no gain · An average above the capacity — a queue only moves load in time; add capacity or shed work
- Trade-offs
- Sized for the average, but work that arrives in a peak finishes later · The backlog drains only with spare capacity — the bigger the peak, the longer the wait · The queue must hold messages safely — durability, size and retention limits
- Requires
- Async work with a queue
3 objectives · 6 misconceptions
- explain Explain how a queue lets a service with a fixed capacity, such as a provider with a send cap, work at its own steady rate — sized for the average, not the peak — and why a consumer that is down then only delays the work.
- predict Predict, from a run's rates, the backlog a peak leaves and when it clears — new work keeps arriving, so only the capacity left over from normal traffic works it off.
- choose Choose between leveling (accept all the work, finish it later) and refusing the excess (rate limiting, backpressure), from how long the work can wait and whether the average fits the capacity.
- Not true: Even with a queue in front, the service has to be sized for the peak.
- Not true: Leveling the load costs nothing — the work gets done just as soon.
- Not true: After a peak, the backlog drains at the workers' full rate.
- Not true: If the worker behind the queue is down, the service that posts to the queue fails too.
- Not true: A queue lets you send to a provider faster than its cap.
- Not true: Load leveling turns away the requests above the service's capacity.