Case
Facebook, 2010: the fix that flooded the database
One configuration change sent every client to the database at once. The database buckled, and its errors made it worse. How do you stop a loop like that?
15 minMediumMembers lesson
Membership isn’t open yet. The free lessons are open — 16 of them.
What you'll be able to do
- 1Predict the database load spike when a hot key expires under heavy traffic.
- 2Choose a mitigation: request coalescing, jittered TTLs or early refresh — or a brake (rate limit, circuit breaker, load shedding) in front of the database.
- 3Predict what happens to the database right after the cache is flushed or restarted.
Inside the lesson
- steps
- 7
- questions
- 4
- simulations
- 3
You guess first, then run the system and see if you were right.
Step 1 · The setup
On 23 September 2010 Facebook was down for many people for about two and a half hours — what Facebook called its worst outage in over four years. A database cluster was overwhelmed — and even after the original problem was fixed, the queries kept coming. Why?
Steps 2–7 are for members