Poison messages & dead-letter queues
Poison messages & dead-letter queues
Silver
Redelivery saves a message from a crash, but a message that fails every time — malformed, or pointing at something that no longer exists — comes back forever, using worker time and calls on every round. A delivery limit stops the loop. After a set number of attempts the queue moves the message to a dead-letter queue, where it waits instead of being lost. A DLQ is a signal, not a bin. Alarm on any message landing there, find and fix the cause, then replay the messages slowly into consumers that can take a repeat.
- Use when
- Any queue with redelivery — a message that always fails must not circle forever · Keeping failed messages for diagnosis and a later replay instead of dropping them · The last step after a consumer's own retries with backoff are used up
- Avoid when
- Work that must stay in strict order — moving one message aside changes the meaning of the ones after it · As a place to forget failures — without an alarm and an owner, a DLQ is silent data loss
- Trade-offs
- A low limit clears poison fast but dead-letters good messages during a short outage · A high limit rides out outages but lets poison burn more attempts first · Replaying fast clears the DLQ sooner but can swamp the consumers and the services behind them
- Requires
- Idempotent consumer
3 objectives · 6 misconceptions
- predict Predict what a message that always fails does to a queue and its workers without a delivery limit, and what changes with a limit and a dead-letter queue.
- choose Choose a delivery limit and dead-letter retention — enough attempts to ride out a short outage, few enough that a poison message leaves soon, and kept longer than the source queue keeps messages.
- explain Explain how to run a dead-letter queue — alarm on its depth, find the cause, fix it, then replay slowly into idempotent consumers — and why order-sensitive work shouldn't use one.
- Not true: A message that keeps failing will get through if the queue keeps retrying it.
- Not true: Messages in a dead-letter queue are dealt with and can be left there.
- Not true: A delivery limit of one is safest, because it moves bad messages out fastest.
- Not true: Only malformed messages ever end up in a dead-letter queue.
- Not true: Once the bug is fixed, replaying the whole dead-letter queue at full speed is the right move.
- Not true: A dead-letter queue is fine for work that must be processed in order.