Sharding a database
Read replicas took the feed, but every write still lands on one primary, and it is full. What changes when Pixel splits its users across several databases — and what gets harder?
13 minMediumStation: ShardingMembers lesson
Membership isn’t open yet. The free lessons are open — 16 of them, including the whole first leg of the Scale Line.
- Uploads, likes, commentsUsers
- Load balancer
- Web server 1Web API
- Web server 2Web API
- Web server 3Web API
- Primary databaseDatabase
What you'll be able to do
- 1Explain why shards add write and storage capacity where read replicas add only read capacity, and why each shard still needs replicas of its own.
- 2Predict each shard's share of the requests, and what a query without the shard key costs, for a given shard key and traffic.
- 3Choose a shard key that keeps most requests on one shard and spreads the load evenly — and reject one that changes, has few values, or scatters the main queries.
Inside the lesson
- steps
- 12
- questions
- 7
- simulations
- 4
You guess first, then run the system and see if you were right.
Step 1 · The setup
Pixel's sister app survived its viral day; Pixel itself kept growing. Read replicas carry the feed, a queue sends the notifications, and the web tier scales itself. But every upload, like and comment is a write, and every write still lands on the one primary database. At the replicas stop there were 30 writes a second; since going viral there are 300.
Watch the primary's CPU bar. (The feed reads are on the replicas and left out of these runs.)
Steps 2–12 are for members
More on the Scale Line
- Latency & percentilesOpen8 min
- Server capacity & scaling upFree with an account10 min
- Split app and database tiersFree with an account10 min
- Load balancerFree with an account10 min
- Horizontal scaling with a stateless tierFree with an account10 min
- Cache-asideFree with an account10 min
- Object storage for user filesMembers10 min
- A CDN for the files everyone wantsMembers12 min