Deliver once

The brief

Notify sends a push for every Pixel post: the Notify API puts it on the send queue, three senders take it off and call the push provider. A quick first version has each sender confirm a message the moment it takes it off the queue, and push whatever it gets — so a push the provider rejects is simply gone. Tonight runs 80 seconds: 200 posts a second until second 40. From second 10 to 30 the provider is down and rejects every push. The queue hands out about 3 messages in 100 a second time. And until second 30, 5 notifications a second name a device that was deleted; the provider rejects those every time. The team’s dead-letter queue is already set up: a message that runs out of tries goes there. Make sure every post’s push reaches the provider effectively once — none lost, none sent twice — within a budget of $350 a month. Where the deleted-device notifications end up isn’t scored in this version, but watch them. In this sim, setting the send queue’s max deliveries — its delivery limit — is how senders confirm only after the provider accepts: a rejected push then goes back to the queue’s tail at once, up to that many tries, and each round takes a few seconds. A sender’s check for repeats skips the queue’s spare copies perfectly. The providers, the cap and the router in front of them are outside Notify’s control, and so are the dead-letter queue and how often the send queue hands out a copy.

What it checks

1. 1Explain at-most-once, at-least-once and exactly-once by when the consumer confirms a message relative to doing the work, and what each one loses or repeats when a worker fails.
2. 2Explain how a stable idempotency key and a record of handled keys, written together with the work in one transaction, turn at-least-once delivery into effectively-once processing — and why checking first and recording afterwards leaves a gap.
3. 3Choose a delivery limit and dead-letter retention — enough attempts to ride out a short outage, few enough that a poison message leaves soon, and kept longer than the source queue keeps messages.

Brush up first

• Delivery guarantees: at most, at least, exactly once
• Idempotent consumer
• Poison messages & dead-letter queues

More practice

1. Keep it aliveOpen8 min
2. Launch dayFree with an account20 min