Design a notification system
A notification system is two systems wearing one coat: a policy engine that decides whether a message is still worth sending, wrapped around a delivery engine that fans one event out to millions. Get the order wrong and you build a very efficient spam cannon.
The hinge: fanout × cost makes synchronous sending impossible
Put the two forcing numbers on the table immediately, because both kill the naive design.
Fanout. A celebrity posts once and 10M followers should be notified. That is one API call from the product, expanding into 10M sends. Suppose your fleet of provider adapters can push 50,000 sends/s in aggregate (a generous number — APNs/FCM throughput plus connection limits). Draining the fanout takes:
10{,}000{,}000 / 50{,}000 = 200 \text{ s} \approx 3.3 \text{ minutes}
Three minutes of continuous work for one event. If the producer (the post service) tried to do this synchronously inside its request, it would block for 3.3 minutes, hold a connection the whole time, and fall over the instant a second celebrity posts. The fanout must land in a queue and drain in the background. This is not an optimization — it is the only shape that survives.
Cost. Now layer in money. Push and in-app are effectively free per message; email is fractions of a cent; SMS is real money — roughly $0.005 per segment. Mis-fire one campaign to those 10M users over SMS:
10{,}000{,}000 \times \$0.005 = \$50{,}000
Fifty thousand dollars for one policy bug — a missing quiet-hours check, a consent flag ignored, a dedupe key that didn't dedupe. In a feed system a bug shows you a stale post. Here a bug is a line item on the finance review. This is why policy is not a "feature" bolted on the side; it is the part of the system most worth getting right, and it must run where it can still prevent the spend.
1. Clarify the contract
Treat the prompt as a product contract before it is a box diagram. The system must:
- Accept a notification intent from a product service ("user X liked your photo", "your code finished building") — not a raw provider call.
- Deliver over multiple channels — push, email, SMS, in-app — each with its own provider, quota, latency, and cost.
- Respect user preferences, quiet hours, and legal consent (CAN-SPAM, GDPR, TCPA) — and respect them before spending.
- Deduplicate the same logical event so a retried producer or a re-delivered queue message doesn't double-notify.
- Retry transient failures without spamming, and give up gracefully on permanent ones.
- Emit delivery / open / failure events for analytics and future suppression.
And explicitly what it need not do: it is not the system of record for the event itself (the post lives in the feed service), it does not guarantee ordering across channels, and it does not promise exactly-once delivery — only effectively-once (at-least-once + idempotency), which we derive below.
2. Data model & API
The API expresses the product operation and hides the queues, shards, and providers behind it.
| API / operation | Why it exists |
|---|---|
POST /notifications {user|segment, type, payload, dedupe_key} | The one ingest path. Producers send intent; the system owns channel/policy/timing. dedupe_key is the producer's promise of identity. |
register_device(user, channel, token) | Device tokens are mutable and expire; this keeps the routing table fresh. |
update_preferences(user, channel, policy) | Opt-outs, quiet hours, frequency caps — the policy data that gates everything. |
The core tables: notification_event (the intent + dedupe key), template (versioned), user_preference (per channel: enabled, quiet-hours window, consent flags, frequency cap), device_token (user → channel → token, with last-seen), and delivery_attempt (event × recipient × channel → state, for idempotency and analytics). The data model is the architecture: if preference ownership is vague, every downstream decision is vague.
3. Linearized design — follow one event through
Walk the event in the order it actually moves; the bottleneck exposes itself.
- Ingest. A product service POSTs a notification intent to the notification service. The service writes it durably and acks fast — it does not send anything inline. For a segment ("all followers of X"), it stores the segment reference, not 10M rows.
- Policy evaluation (the gate). Before any channel work, evaluate per recipient: is this channel enabled? Are we inside their quiet hours? Is there valid consent for this message class? Has their frequency cap been hit? What channel does priority dictate? Recipients that fail policy are dropped here — costing nothing. This is where the $50k bug is prevented.
- Dedupe. Check the dedupe store for
(user, dedupe_key). Seen recently? Suppress. This collapses producer retries and "liked, then unliked, then liked again" storms. - Render. Render the template against a stable snapshot of the data, so a retry days later produces the same message it would have at send time (no semantic drift).
- Fan out to per-channel queues. Push, email, and SMS each have their own queue with independent rate limits and consumer pools. A slow SMS provider cannot back up the push pipeline.
- Provider adapter. Each channel's workers call the provider (APNs/FCM, SES/SendGrid, Twilio) behind a circuit breaker, carrying an idempotency key so a redelivered queue message doesn't re-send. The adapter normalizes the response into {success | retryable | permanent}.
- Feedback. Record the attempt; ingest delivery/open/bounce webhooks. Permanent failures (dead token, hard bounce) feed back into suppression.
4. Deep dives
4.1 Policy before delivery — or you've built a spam cannon
The single most common junior mistake is to draw ingest → queue → provider and bolt preferences on as a filter near the end. That ordering is wrong for both reasons that define this system. Cost: if you check quiet hours after rendering and enqueuing 10M SMS messages, you've already done the expensive work — and if the check itself has a bug, you've already spent the $50k. Policy must run where it can still prevent the spend, which is upstream of the channel queues. Trust: a user who gets one notification at 3am they explicitly disabled will disable all notifications and never come back; that lost engagement dwarfs the cost of the single send.
So preferences, quiet hours, consent, and priority are first-class data models, not config flags. Priority deserves emphasis: a security alert ("new login from an unknown device") overrides quiet hours and frequency caps; a "someone you may know joined" does not. Encoding priority as a property of the message class lets the gate make that call deterministically. The rule of thumb: every byte of policy you can evaluate before enqueuing saves you a byte of cost and a unit of user trust.
4.2 At-least-once queues → duplicates are normal → idempotency is mandatory
This is the delivery-semantics heart of the case. Queues that survive crashes give you at-least-once delivery: a worker can pull a message, send it, and die before acking — so the broker redelivers it, and the user gets two pushes. You cannot wish this away; true exactly-once across a network and a third-party provider is impossible (the provider might have received your request and only the ack got lost). What you can build is effectively-once = at-least-once transport + idempotent consumers. This is DDIA Ch. 11's idempotent-consumer pattern, and the concept is taught in full in C27 (delivery semantics) — here we apply its two distinct keys:
- Dedupe key — at the user-event level. The producer's
dedupe_keyidentifies the logical event ("post 9981 liked by user 42"). The policy stage records it; a second arrival of the same logical event is suppressed. This protects against producer retries and event storms. - Idempotency key — at the provider adapter. A per-send key (event_id × recipient × channel) stored insert-if-absent before the provider call. If a queue message is redelivered, the adapter sees the key already exists in a terminal state and skips the resend. This protects against queue redelivery. Many providers also accept their own idempotency token (Twilio, Stripe-style) — use it when available so even a lost-ack retry is dedup'd provider-side.
Two keys, two layers, two failure sources. Conflating them — using one dedupe at ingest only — leaves the window between "message pulled from queue" and "provider acked" unprotected, which is exactly the window that produces the double-push in production.
4.3 Batching and segmentation — making a 10M fanout controllable
If a celebrity-post event expands into 10M individual queue messages at ingest, you've created 10M rows you cannot easily pause, cancel, or reprioritize, and you've put a 200-second write storm on the broker. Instead, fan out lazily in batches: store the event as a segment job ("notify followers of X, batch by 10k"), and let a fanout worker materialize recipient batches and feed them to the channel queues at a controlled rate. This buys three properties the flat design can't:
- Pause / cancel. A campaign found to be buggy (the $50k case) can be halted after batch 50 of 1000 — you've sent 500k, not 10M. The flat design has already committed all 10M.
- Rate-limit per channel. The SMS queue drains at 200/s while push drains at 50k/s; batching lets each channel pull at its own pace from one logical job.
- Reprioritize. A high-priority transactional notification can jump ahead of a low-priority marketing fanout that's mid-drain.
This is the C08 fanout-on-write tension at notification scale: you trade a little ingest latency (you don't materialize all recipients immediately) for enormous operational control over the long tail of the send.
5. Trade-offs
| Choice | Buys | Costs | Choose when |
|---|---|---|---|
| Synchronous send vs queued send | Immediate delivery confirmation to the producer. | Producer blocks on provider latency; cannot survive fanout (3.3 min for 10M). | Only tiny, low-fanout internal notifications. |
| User-level dedupe vs event-level dedupe | User-level cuts more spam (collapses bursts). | May suppress genuinely distinct, useful events. | Like/comment bursts → user-level; transactional → event-level. |
| One queue vs per-channel queues | Single queue is simpler to operate. | An SMS provider outage backs up and blocks push/email. | One queue only for small, single-channel systems. |
| Eager fanout vs batched/segmented fanout | Eager is simplest; every recipient is a row immediately. | No pause/cancel/reprioritize; 10M-row write storm at ingest. | Eager only when max fanout is small and bounded. |
The sharpest one is per-channel queues. It looks like premature complexity — why not one queue with a channel field? Because the channels have wildly different failure and rate profiles: SMS at 200/s through a flaky third party versus push at 50k/s through a robust one. With a shared queue, a Twilio outage means SMS messages pile at the head and starve push consumers (head-of-line blocking), so a marketing SMS hiccup silently delays your 2FA pushes. Independent queues give each channel its own backpressure, rate limit, and circuit breaker — isolation is the whole point.
6. Failure modes
| Failure | Mitigation |
|---|---|
| Provider outage (e.g. APNs down) | Circuit-break the adapter, retry with exponential backoff + jitter, fall back to another channel where appropriate — and drop notifications whose value has expired (see Q&A). |
| Invalid / expired device tokens | Treat as permanent failure; cull from the routing table and refresh on next app launch. Don't retry a dead token. |
| Notification storm (runaway producer, retry loop) | Per-user and global frequency budgets with priority shedding — drop low-priority first (foundation lesson 15). |
| Template bug | Version templates; canary high-volume campaigns to 1% before full fanout; the batched fanout lets you abort mid-send. |
| Queue redelivery → double send | Idempotency key at the adapter (4.2); the duplicate is suppressed before the provider call. |
7. Interview Q&A
- APNs goes down for 20 minutes — what happens to 2M queued pushes? (senior answer) First, the circuit breaker on the push adapter trips after a failure threshold, so workers stop hammering a dead provider and instead fast-fail, leaving messages in the queue. Retries use exponential backoff with jitter so all 2M don't stampede the instant APNs recovers (thundering herd). Crucially, I drop notifications whose value has expired: a "your build finished" or "X is live now" push is worthless 20 minutes late, so it carries a TTL and is discarded rather than delivered stale. High-value, time-insensitive ones (a security alert) survive the wait or fall back to another channel (SMS/email). The mistake is queueing all 2M forever and dumping them on the user when APNs returns — a 20-minute-old notification flood is itself an incident.
- Why must policy run before delivery, not as a filter at the end? (senior answer) Two reasons: cost and trust. Checking quiet hours after enqueuing 10M SMS means you've already risked the $50k spend before the check runs; and the whole value of the system is not sending unwanted messages, so the gate belongs upstream where it can prevent work. Preferences, consent, and priority are first-class data, not a trailing filter.
- You have at-least-once queues. How do you avoid double-notifying? (senior answer) You can't get true exactly-once across a third-party provider, so you build effectively-once: at-least-once transport plus idempotent consumers. Two keys — a dedupe key at the user-event level (collapses producer retries/storms) and an idempotency key at the provider adapter, insert-if-absent before the send (collapses queue redelivery). That's DDIA Ch. 11's idempotent consumer; the concept anchor is C27.
- A celebrity with 10M followers posts. Walk the fanout. (senior answer) Ingest stores a segment job, not 10M rows, and acks immediately. A fanout worker materializes recipient batches (say 10k) and feeds them to per-channel queues at a controlled rate; at 50k/s push drains in ~3.3 min. Batching means I can pause or cancel mid-fanout if the campaign turns out buggy — I've sent 500k, not 10M.
- Why per-channel queues instead of one queue with a channel field? (senior answer) Isolation. SMS (200/s, flaky third party) and push (50k/s, robust) have totally different rate and failure profiles. A shared queue means an SMS outage causes head-of-line blocking that starves push — so a marketing SMS problem delays your 2FA codes. Separate queues give each its own backpressure, rate limit, and circuit breaker.
- Should you key the limit / dedupe on IP, user, or device? (senior answer) On the user for preferences and frequency caps (the policy unit), and on the device token for actual push routing (a user has many devices). The dedupe key is on (user, logical-event), and the provider idempotency key is on (event, recipient, channel) so the same event to the same person on two channels isn't wrongly collapsed.
- How do you stop a runaway producer from spamming everyone? (senior answer) Per-user and global frequency budgets enforced at the policy gate, with priority shedding — low-priority marketing drops first, transactional and security survive. This is foundation lesson 15's admission control applied to notifications; the limiter protects users and the SMS budget, not just the backend.
- How do delivery/open feedback events close the loop? (senior answer) Provider webhooks (delivered, bounced, opened, unsubscribed) flow into the feedback store. Hard bounces and dead tokens become suppression rules so we stop wasting sends and money; open rates feed engagement; unsubscribes update preferences. Without this loop you keep paying to send into the void and your sender reputation rots.
Related foundation lessons
This case leans on several foundations, each for a specific reason. Lesson 11 (async messaging) is the backbone — the per-channel queues and log-based brokers that make the 3.3-minute fanout survivable (DDIA Ch. 11's stream processing and log-based message brokers). Lesson 13 (fault tolerance) supplies the circuit breakers and exponential-backoff-with-jitter that handle the APNs-down scenario (and DDIA Ch. 8's provider timeouts and retries). Lesson 16 (observability) gives the delivery/open/bounce metrics and SLOs that tell you whether the system is healthy or quietly burning money. And C27 is the anchor for delivery semantics — at-least-once vs exactly-once and idempotent consumers — which we only apply here.