Pub/sub fan-out
One new post, three services that each need it. How does Notify reach them all without tying itself to any of them — and what does that cost?
12 minMediumStation: Pub/sub fan-outMembers lesson
Membership isn’t open yet. The free lessons are open — 16 of them.
- Load leveling
- Competing consumers
- Pub/sub fan-out
- Notify’s big night
What you'll be able to do
- 1Explain how a topic gives every subscriber its own copy of each message where a queue gives each message to one consumer, and why adding a subscriber needs no change to the publisher.
- 2Predict the message volume and each subscriber's backlog when one publish rate fans out to several subscribers that work at different speeds.
- 3Choose a topic or a queue from whether a message is a job for one worker or news for anyone interested — and combine them, with a queue per subscriber and competing workers on it.
Inside the lesson
- steps
- 11
- questions
- 6
- simulations
- 4
You guess first, then run the system and see if you were right.
Step 1 · The setup
Notify's senders get pushes out — at about 80 posts a second (each post is still one notification to Notify), well inside the provider's cap. Now each new post must also reach people by email and land in the app's inbox.
The API could call push, email and the inbox itself. Then it must know where each lives, deliver to each and handle each one's failures — and every new consumer means editing it. Putting an email worker and an inbox worker on the send queue doesn't work either: workers on one queue share its messages, so each post would reach only one of the three.
So Notify publishes each post once, to a topic. Push, email and the inbox each subscribe, with a queue of their own and a worker on it.
Steps 2–11 are for members