Rate limiting
Rate limiting
Silver
A rate limit caps how many requests each client may send in a time window — commonly a token bucket with a steady rate and a burst allowance. Requests over the limit get a quick "too many requests" answer, which protects the service and shares it fairly between clients.
- Use when
- Shared APIs where one client's surge must not crowd out the others · Guarding against floods — buggy clients, retry storms, abuse · Pacing your own calls to a provider that enforces a limit
- Avoid when
- When the bottleneck is concurrency or costly work rather than request rate — use concurrency limits or bulkheads · As the only protection — a limit set above real capacity still lets the service overload
- Trade-offs
- Fair and predictable, but a client over its quota sees errors even when the service has room · Bigger bursts absorb natural spikes but weaken the protection · Limits need load-tested values and revisiting as the workload changes
- Requires
- Load shedding, Retries with backoff and jitter
3 objectives · 8 misconceptions
- explain Explain how a token bucket's refill rate and burst size decide which requests get through.
- predict Predict what each client sees when one client floods a service behind a single shared limit, and behind per-client limits.
- choose Choose a limit's key, rate and burst from tested capacity, and how a client should react to a 429.
- Not true: One overall limit for the whole service protects everyone as well as a limit per client does.
- Not true: A bigger burst allowance only makes a limit friendlier.
- Not true: Once a rate limit is in place, the service can't be overloaded.
- Not true: A throttled request is a server failure, like a 500.
- Not true: After a 429, the client should resend the request right away.
- Not true: The quickest way through a rate-limited API is to send everything at once and resend whatever is refused.
- Not true: A client hitting its limit means the service is out of capacity.
- Not true: A client that goes over its limit is cut off completely.