Z3 · Data & Storage

Sharding

Gold

Sharding splits one database's rows across several databases, called shards. Each shard has the same tables but holds only its own slice of the data, picked by a shard key such as the user id. Unlike read replicas, which copy everything, each shard takes only its own share of the writes and storage, so writes can grow with the number of shards. The price is that every request needs its shard key, and a question that spans shards — or a transaction across them — gets slow and hard.

Use when: Writes or stored data outgrow the biggest database server you can buy · Read replicas and caches already took the reads, and the primary still saturates on writes · Most requests naturally belong to one owner — a user, a tenant, an account — that can be the shard key
Avoid when: The data and traffic still fit one server, even with expected growth — scale up instead · The bottleneck is reads — use read replicas or a cache · The main queries join or aggregate across all the data, or need transactions across many owners · A managed database already shards for you — use its partition key instead of building your own layer
Trade-offs: Write and storage capacity grow with shards, but every request must carry the shard key · Queries and transactions across shards are slower and harder; most systems avoid cross-shard transactions · A partial outage is possible — one shard down takes only its slice of the data · Backups, schema changes and monitoring now happen per shard, for the life of the system
Requires: Read replicas, Horizontal scaling with a stateless tier

3 objectives · 7 misconceptions

explain Explain why shards add write and storage capacity where read replicas add only read capacity, and why each shard still needs replicas of its own.
predict Predict each shard's share of the requests, and what a query without the shard key costs, for a given shard key and traffic.
choose Choose a shard key that keeps most requests on one shard and spreads the load evenly — and reject one that changes, has few values, or scatters the main queries.

Not true: Each shard holds a full copy of the data, like a read replica.
Not true: When writes outgrow the database, adding read replicas will fix it.
Not true: A query that needs every shard costs about the same as one that needs a single shard.
Not true: Any key whose values are all different, like an upload time, makes a good shard key.
Not true: If the shard key turns out badly, you can switch to a better one later without much work.
Not true: When one shard fails, the whole database is down.
Not true: Every growing app should shard its database as early as possible.

Learn

Sharding a databaseMembers~13 min

Practice

Celebrity dayMembershard

Check

Station test-out80% to pass

Stories here

Discord: billions of messages, then trillionsMembersstory

Neighborhood

builds onRead replicas Horizontal scaling with a stateless tier

→Sharding→

unlocksConsistent hashing Multi-region

contrast withRead replicas

On linesScale Line