Hot keys
Hot keys
Gold
A good hash spreads keys evenly over the shards, but not requests. When a few keys get much of the traffic — a celebrity's profile, today's date — the shard that owns them is overloaded while the others sit idle, even though together they have room. Adding shards doesn't help, because every request for one key goes to the one shard that owns it. The fixes work on the key itself — cache its reads, split its writes over several sub-keys, or give it a server of its own — and each has a cost.
- Use when
- One user, item or date draws far more traffic than the rest — a celebrity, a viral post, today's partition · One shard is busy or throttled while the others are idle · A key that only grows (a timestamp, a counter) sits at the front of a range-partitioned key
- Avoid when
- Load is high on every shard — that is a capacity problem, not a hot key; add shards or bigger ones · The busy shard is busy because of its data size, not its requests — rebalance the data instead
- Trade-offs
- Caching hot reads is cheap and effective, but the cached copy can be stale · Splitting a key into N sub-keys spreads its writes, but reading it all back means N reads and a merge · Giving a hot key a server of its own works up to that server's limit, and someone must spot and move it
- Requires
- Consistent hashing, Cache-aside
3 objectives · 11 misconceptions
- explain Explain why adding shards doesn't cool a hot key, since every request for a key goes to the one shard that owns it.
- predict Predict the busiest shard's load from the share of traffic its hottest keys draw, and whether it goes over that shard's capacity.
- choose Choose a fix for a hot key — cache its reads, split its writes over sub-keys, or isolate it — and name what each one costs.
- Not true: Adding more shards fixes a hot key.
- Not true: A good hash function spreads the requests evenly, whatever the traffic looks like.
- Not true: If the shards together have capacity for the total traffic, no shard can be overloaded.
- Not true: A hot shard is one that holds more data than the others.
- Not true: Splitting a hot key into several sub-keys costs nothing when reading it back.
- Not true: A managed database rebalances by itself, so one very busy item can never be throttled.
- Not true: A hot key is just another name for a cache stampede.
- Not true: A hot shard is one whose slice of the ring is too big, so evening out the slices fixes it.
- Not true: A hot spot always comes from a popular item, never from how the key is designed.
- Not true: Sub-keys made from one key (its id plus a suffix) all land on the shard that owned the original key.
- Not true: Splitting a hot key into sub-keys just moves the hot spot to another shard.
Stories here
Neighborhood
contrast withCache stampede