Designing a Notification System That Fans Out to Push, Email, and SMS

One event happens. A user gets mentioned in a comment. Now you might need to send a push notification, an email, and an in-app badge, and you have to respect that this person muted email but still wants push. Multiply that by millions of events per hour and the naive version falls apart fast. This is one of those systems that looks trivial from the outside and turns into a mess the moment you try to make it reliable.

The Core Problem

A notification system is a fanout engine with three hard requirements pulling against each other: deliver fast, never lose a message, and never spam the user. The trouble is that the producers of events (comments, follows, payments, alerts) have no idea how a given user wants to be reached, and the delivery channels (APNs, FCM, an email provider, an SMS gateway) each fail in their own way and at their own speed. If you couple event producers directly to channel delivery, one slow provider stalls the whole thing. So the first decision is to decouple. Producers write an event. A separate pipeline decides who gets notified and how.

Key Design Decisions

Split into ingestion, preference resolution, and delivery. An event lands on a queue. A worker resolves the recipient set and, for each recipient, looks up their channel preferences and quiet hours. Only then does it enqueue per-channel delivery jobs. Keeping these as separate stages means a backed-up SMS gateway never blocks push.

Treat preferences as a first-class store, not an afterthought. For each user you need per-category, per-channel opt-in flags, a timezone, quiet hours, and a digest setting. Category matters because "security alert" should ignore quiet hours while "someone liked your post" should not. This table gets read on every single notification, so it wants aggressive caching.

Make delivery idempotent. Queues give you at-least-once delivery, which means the same job can run twice. If you are not careful the user gets two identical emails. Attach a stable notification id and dedupe on it at the channel worker, or use the provider's idempotency key where one exists.

Rate limit and batch per user. If someone triggers fifty events in a minute, you do not send fifty pushes. You collapse them. A short aggregation window (say, a few minutes) lets you turn "12 new comments" into one notification. This is a product decision as much as a technical one, but the mechanism lives in the pipeline.

Handle channel failure with retries and dead-letter queues. APNs returns a bad-token response and you must mark that device token dead so you stop trying. A transient 500 from an email provider should retry with backoff. A message that fails every retry goes to a dead-letter queue for inspection, not into the void.

The Read Side People Forget

In-app notifications need storage and a fast unread count. A common approach is a per-user notification feed in a store that handles high write volume, plus a cached unread counter you increment on write and reset on read. Do not compute the unread count with a COUNT query on every page load.

How the Real Systems Do It

Large platforms run this as a pipeline on top of a log like Kafka. Events are produced to a topic, fanout workers consume and expand them into per-recipient work, and channel-specific consumers own the last mile to APNs, FCM, and email vendors. Device token lifecycle management is its own subsystem because tokens expire and rotate constantly, and sending to dead tokens wastes quota and hurts your sender reputation. Digesting and quiet-hours logic tends to live in the fanout stage so that no channel worker has to know about it.

The theme across all of it is the same: decouple the event from the delivery, make every step retryable and idempotent, and put the user's preferences at the center instead of bolting them on at the end.

I wrote the full breakdown, with the data model, the fanout diagram, and the interview talking points, here: https://www.systemdesign.academy/interview/design-notification-system