Codex
44 concepts
Networking & Internet
1 conceptsCompute & Execution
4 conceptsServers & capacity
- Queueing & Little's lawRequests arrive, wait their turn and get served. Little's law ties three averages together — the number of requests inside a system equals the rate they arrive times how long each one stays — for a queue, a server, a pool or a whole service. A queue only settles while arrivals stay below what its server can serve.Not met yet
- Capacity mathCapacity math turns a peak load into the resources to provision — what each resource carries at that peak (CPU, connections in use, the database's connection limit), kept below its knee, with a spare for a lost server. The first resource to run out is the bottleneck, and a load test checks the estimate.Not met yet
- Connection poolsOpening a database connection is slow and costly, so a service keeps a pool of open connections and lends one to each request for as long as its database work takes. The pool caps how many requests talk to the database at once — connections in use ≈ requests per second × how long each holds one — so a slower database can drain the pool at unchanged traffic.Not met yet
- Server capacity & scaling upA server can only do so much work per second. As utilization nears 100%, requests queue and latency shoots up. A bigger machine buys headroom until the biggest one runs out.Not met yet
Scalability & Performance
7 conceptsSpreading load
- Load balancerA load balancer spreads incoming requests across several servers and routes around unhealthy ones, so capacity can grow by adding machines.Not met yet
- Hot keysA good hash spreads keys evenly over the shards, but not requests. When a few keys get much of the traffic — a celebrity's profile, today's date — the shard that owns them is overloaded while the others sit idle, even though together they have room. Adding shards doesn't help, because every request for one key goes to the one shard that owns it. The fixes work on the key itself — cache its reads, split its writes over several sub-keys, or give it a server of its own — and each has a cost.Not met yet
Scaling strategies
- AutoscalingAutoscaling adds servers when a metric such as CPU crosses a threshold and removes them when load falls, so capacity follows traffic instead of being sized for the peak all day.Not met yet
- Horizontal scaling with a stateless tierRun many identical, stateless app servers behind a load balancer. Capacity grows by adding servers, and losing one is survivable. State moves to shared stores.Not met yet
- Split app and database tiersMoving the database onto its own server lets the app and data tiers be sized, scaled and replaced independently — the first step from one box to a system.Not met yet
- Multi-regionRunning the app in several cloud regions puts it near users on other continents and lets it survive a whole region going down. Copying the servers is the easy part; the data decides the design. Each region can keep a copy that is updated asynchronously — fast local reads and writes, but stale reads and conflicting writes between regions. Or replication can be synchronous, so every write waits for the far region. Or each user's data can live in one home region. Failover also needs spare capacity, because the surviving regions must carry the lost region's users.Not met yet
Caching & Delivery
7 conceptsCache fundamentals
- Memory vs disk vs networkReading from memory takes nanoseconds, from an SSD tens to hundreds of microseconds, from a spinning disk milliseconds — and a network round trip adds anything from a millisecond or two inside one region to well over a hundred between continents. The steps are not small, which is why a copy of hot data kept in memory, close to the code that reads it, pays off.Not met yet
- Cache basics: hit ratio, TTL, evictionA cache keeps copies of often-used data in fast memory. Its value comes from the hit ratio; TTLs and eviction decide what stays and how stale it may get.Not met yet
Cache strategies
- Cache invalidationInvalidating means removing cached copies when the data behind them changes. Deleting the entry is simpler and safer than updating it in place, but the delete must reach every copy — each server's own cache, the shared cache, every key built from the changed data — and beat any slow read still carrying the old value. Whatever it misses stays stale until its TTL runs out; invalidating everything at once trades stale reads for a flood of misses.Not met yet
- Cache stampedeWhen a popular cached item expires, or the whole cache is flushed, many requests miss at once and hit the database together — which can overload it.Not met yet
- Cache-asideThe app checks the cache first; on a miss it reads the database and stores the result in the cache. Writes go to the database and expire or remove the cached copy.Not met yet
- CDN (content delivery network)A CDN keeps copies of static content on servers close to users, so most requests for images and assets never reach your origin — and arrive sooner.Not met yet
- Read-through & write-throughIn read-through, the cache itself loads a missing item from the database. In write-through, each write goes to the database and the cache together and is confirmed only once both have it — so a read straight after a write gets the new value, at the price of slower writes and memory spent on data nobody may read. Write-around writes the database alone and leaves the cache to load the item on its next miss.Not met yet
Data & Storage
9 conceptsData modeling
- Relational databases & indexesA relational database keeps data in tables — rows with the same named columns — and links tables by keys, so one fact is stored once and SQL joins rows back together when you read. Without an index, finding a row means scanning the whole table; a B-tree index keeps the column sorted, so the database jumps to the row in a few page reads, even among millions. Indexes are not free — every insert, update and delete must also update each index, and each one takes disk space. A multi-column index helps only queries that use its leftmost columns, and EXPLAIN shows whether a query really uses the index or scans.Not met yet
- Access patterns & denormalizationIn a NoSQL store you design the data from the questions the app will ask, not from the entities alone. First list the access patterns — "the latest 50 messages in a group", "this member's balance" — then choose keys so each one is a single, cheap lookup. The partition key decides which items are stored together; the sort key keeps them in order inside that partition, so "the latest N" or "this month's" is one range read. Data read together is stored together, even if that means copying it — denormalization. Copies make reads cheap and writes expensive — every change must reach every copy, and each copy takes storage — so normalize what changes often and copy what is read often.Not met yet
- NoSQL families"NoSQL" names several different kinds of database that are not built around relational tables. A key-value store fetches one value by its key, very fast and nothing else. A document store keeps each record as a JSON-like document that can nest lists and objects, so one read returns a whole object. A wide-column store spreads huge, sparse tables across many machines and suits heavy write streams such as events or messages. A graph database stores nodes and the links between them and follows links quickly. Each one gives up some query freedom — ad-hoc joins, transactions across many items — to scale out and stay fast on the queries it was shaped for. NoSQL does not mean no schema, always faster, or no transactions.Not met yet
Partitioning data
- Consistent hashingTo find a key's shard, you hash the key. The simple way — hash mod the number of shards — breaks when that number changes, because almost every key lands on a different shard and the data must move. Consistent hashing puts shards and keys on a ring, and each key belongs to the next shard round the ring. Adding a shard then moves only the keys it takes over — about 1/N of them — and removing one hands on only its own keys. Giving each server many points on the ring (virtual nodes) keeps the slices even.Not met yet
- ShardingSharding splits one database's rows across several databases, called shards. Each shard has the same tables but holds only its own slice of the data, picked by a shard key such as the user id. Unlike read replicas, which copy everything, each shard takes only its own share of the writes and storage, so writes can grow with the number of shards. The price is that every request needs its shard key, and a question that spans shards — or a transaction across them — gets slow and hard.Not met yet
Storage basics
- Databases: the system of recordA database stores data durably and answers queries. It is usually the slowest and hardest part to scale, so most designs revolve around protecting it.Not met yet
- Object storageObject storage keeps files such as photos and videos as whole objects behind an HTTP API, so app servers stay stateless and the files outlive any one server.Not met yet
- Read replicasRead replicas are copies of the database that serve reads, so a read-heavy app can add read capacity while every write still goes to one primary.Not met yet
- Replication lag & read-your-writesRead replicas apply the primary's changes a little later — usually milliseconds to a second, but longer when writes surge, a replica is busy or a transaction is big, and with no fixed upper limit. A read that lands on a replica in that gap returns stale data, such as an expense you just added missing from the list. Read-your-writes is the promise that you always see your own changes; apps get it by sending a user's reads to the primary for a short while after they write, or by remembering the position of their last write and reading only from a replica that has caught up to it. Synchronous replication closes the gap but makes every commit wait for the replica.Not met yet
Messaging & Events
4 conceptsQueues & background work
- Async work with a queueA queue lets the web tier hand slow side work, like notifying followers or sending email, to background workers and answer the user right away; the workers catch up at their own pace.Not met yet
- Competing consumersSeveral workers take messages from one queue, and each message goes to whichever worker is free next. Adding workers raises throughput — until something they all share, such as an outside provider's cap or the database, becomes the bottleneck. Messages finish out of order, and a worker that crashes mid-message hands it back — it becomes visible again and another worker takes it, so a message can be processed twice.Not met yet
- Load levelingA queue in front of a service with a fixed capacity — such as an outside provider that caps how fast you may send — lets that service work at its own steady rate. Peaks wait in the queue, so the service is sized for the average, not the peak. The price is delay — a peak leaves a backlog that drains only with the capacity left over from normal traffic.Not met yet
- Pub/sub fan-outA publisher sends each message once to a topic, and the topic hands a copy to every subscription — unlike a queue, where each message goes to one consumer. The publisher doesn't know or wait for its subscribers, so adding one needs no change to it. Each subscription usually has its own queue, so a slow or failing subscriber falls behind alone. The cost is volume — N subscribers turn each message into N deliveries.Not met yet
Resilience & Reliability
8 conceptsFault tolerance
- BackpressureWhen a consumer can't keep up, backpressure carries a "slow down" signal back to the producer — typically a bounded queue that refuses new work when it is full — instead of letting an unbounded queue grow its delay and memory without limit.Not met yet
- BulkheadA bulkhead splits a system's resources — servers, connection pools, threads — into separate pools for different workloads or dependencies, so that when one pool fails or is overloaded, the others keep working.Not met yet
- Circuit breakerA circuit breaker watches calls to a dependency. After too many failures it opens and fails fast for a while, then lets a few trial calls through to check whether the dependency has recovered.Not met yet
- Load sheddingWhen a server nears its capacity it starts turning away the excess requests with a fast, cheap error, so the requests it does accept still finish quickly — instead of slowing down for everyone until most requests time out.Not met yet
- Rate limitingA rate limit caps how many requests each client may send in a time window — commonly a token bucket with a steady rate and a burst allowance. Requests over the limit get a quick "too many requests" answer, which protects the service and shares it fairly between clients.Not met yet
- Retries with backoff and jitterRetrying a failed call turns a brief glitch into a success — but every retry is extra load on a dependency that may already be struggling, so retry only what is safe to repeat, a few times, waiting longer each time with some randomness.Not met yet
- TimeoutsA timeout caps how long a caller waits for a dependency. Without one, a slow dependency ties up threads and connections until the caller fails too.Not met yet
Operations & Cost
4 conceptsCloud cost
- How cloud billing worksThe cloud bills by meters, and each kind of service has its own. A server is billed for every second it runs — whether it is busy, idle or stuck. A managed service such as a queue or a storage bucket is billed for what you use — per request, or per gigabyte. A serverless function is billed per call plus the memory it holds for as long as it runs, and costs nothing while nobody calls it. Around these sit smaller charges that are easy to miss — disks, addresses, logs and data sent out to the internet. To read a bill, find the meter behind each line; to estimate a month, multiply the hourly rate by about 730.Not met yet
- Autoscale economicsA fleet sized for the evening peak is paid for at peak size all day, even at 4 a.m. Autoscaling lets the bill follow the traffic instead — you pay for instance-hours, so fewer servers in quiet hours means a smaller bill. How much you save depends on how far the peak is above the average; flat traffic saves almost nothing. Every safety setting has a price — a higher minimum, more headroom and a slower scale-in all cost money, and a maximum caps the bill but turns away anything above it. Known peaks can be scheduled ahead of time, and the steady base that always runs can be bought cheaper with a long-term commitment.Not met yet
- Right-sizingA server is billed for its size, not for how busy it is, so a big machine running at 10% is mostly paying for nothing. Right-sizing means measuring what each resource really uses — CPU and memory, at its busiest moments, over days or weeks — and moving it to the smallest size that still carries that peak with room to spare. Within one instance family, a smaller size usually costs less and runs busier for the same load — so the risk is cutting into the headroom. Done well, it is the cheapest saving there is; done from averages, or from CPU alone, it causes the next outage.Not met yet
- Serverless vs always-onA serverless function is billed per call and for the memory it holds while it runs, so with no traffic it costs nothing. A server is billed every hour it exists, busy or not. At low or bursty traffic the function wins easily; at steady high traffic the per-call price adds up, and there is a request rate above which one always-on server is cheaper. The function's other costs show up as behaviour — after a quiet spell the first calls wait for a cold start, too many calls at once get throttled, and a call that hangs until its timeout is billed for the full time. Keeping functions warm removes cold starts, but brings back a bill for idle time.Not met yet
Progress is saved on this device