Z3 · Messaging & Events

Competing consumers

Silver

Several workers take messages from one queue, and each message goes to whichever worker is free next. Adding workers raises throughput — until something they all share, such as an outside provider's cap or the database, becomes the bottleneck. Messages finish out of order, and a worker that crashes mid-message hands it back — it becomes visible again and another worker takes it, so a message can be processed twice.

Use when: Independent jobs that can run side by side — one notification per message · Work volume that varies, with workers added or removed as the backlog changes · Keeping work flowing when one worker fails
Avoid when: Jobs that must run in a strict order · Work that can't be split into independent messages · A shared downstream that is already the bottleneck — more workers only add load there
Trade-offs: More throughput and redundancy, but order is lost across workers · A crashed or slow worker's message comes back — handlers must cope with doing it twice · The hiding timeout must outlast the work, but not by so much that failed messages wait long to return · An upper limit on workers keeps their combined rate within what downstream can take
Requires: Load leveling

3 objectives · 7 misconceptions

predict Predict a queue's throughput and backlog as workers are added, up to the point where a shared downstream — a provider's cap, a database — becomes the bottleneck.
explain Explain why a crashed worker's message isn't lost — it stays hidden in the queue until the worker reports it done — and why that means a message can be processed twice.
choose Choose how many workers to run and what to scale them on — the backlog per worker against the wait you can accept — and spot work that can't be shared out because it must stay in order.

Not true: Each worker added to a queue adds the same throughput, however many there already are.
Not true: Messages are finished in the order they were sent, however many workers share the queue.
Not true: When a worker crashes in the middle of a message, that message is lost.
Not true: A message handed back after a crash is sent exactly once in the end.
Not true: A message can be processed twice only if a worker crashes.
Not true: Every worker on the queue gets its own copy of each message.
Not true: The total number of messages waiting is the right number to scale workers on.

Learn

Competing consumersMembers~13 min

Practice

Notify’s big nightMembersmedium

Check

Station test-out80% to pass

Neighborhood

builds onLoad leveling

→Competing consumers→

unlocksPub/sub fan-out

contrast withPub/sub fan-out

On linesAsync Line