Poison messages & dead-letter queues
Redelivery saves a message from a crash — but a message that fails every time comes back forever. How does a delivery limit stop it, where do the failures go, and how do you get them back out safely?
13 minMediumStation: Poison messages & dead-letter queuesMembers lesson
Membership isn’t open yet. The free lessons are open — 17 of them.
- Pixel's postsUsers
- Posts for a deleted deviceUsers
- Notify APIWeb API
- Send queueQueue
- Sender 1Worker
- Sender 2Worker
- Sender 3Worker
- Push provider's cap (300/s)Rate limiter
- By deviceRouter
- Push provider (device deleted)Service
- Push providerService
What you'll be able to do
- 1Predict what a message that always fails does to a queue and its workers without a delivery limit, and what changes with a limit and a dead-letter queue.
- 2Choose a delivery limit and dead-letter retention — enough attempts to ride out a short outage, few enough that a poison message leaves soon, and kept longer than the source queue keeps messages.
- 3Explain how to run a dead-letter queue — alarm on its depth, find the cause, fix it, then replay slowly into idempotent consumers — and why order-sensitive work shouldn't use one.
Inside the lesson
- steps
- 12
- questions
- 9
- simulations
- 4
You guess first, then run the system and see if you were right.
Step 1 · The setup
Notify's senders confirm a notification only once the provider accepts it; anything else comes back to the queue and is tried again. That saved every push when a sender crashed. It also has a cost nobody has met yet.
Some users delete the app. Their device is gone, and the provider rejects every push to it. In this sim those pushes are drawn as their own stream — 10 a second for 30 seconds — and their own route, so you can watch them. Nothing limits how often a notification comes back.
Steps 2–12 are for members