Q64 — Design a notification system (push, email, SMS at scale)

What the system must do

Multiple upstream services (order service, payment service, social service) need to notify users via multiple channels: push notifications (iOS/Android), email, and SMS. A user placing an order gets an order confirmation email, a push notification, and possibly an SMS for delivery updates.

Core requirements:

Accept notification requests from multiple upstream services
Route to the right channel(s) per user (based on preferences)
Deliver reliably — a missed payment confirmation is a serious UX failure
Handle channel-specific providers (APNs/FCM for push, SendGrid/SES for email, Twilio for SMS)
Scale to tens of millions of notifications per day
Not block the upstream service — the payment service should not wait for email delivery

The fundamental architecture decision: asynchronous always

The order service cannot call the notification service synchronously and wait. Email delivery through SendGrid takes 50–500ms. FCM/APNs delivery can take seconds. Blocking the order creation flow on notification delivery is unacceptable — and if the notification service is down, the order would fail.

The notification system must be entirely asynchronous. Upstream services publish notification events to a message queue; the notification system consumes and delivers independently (Q42 — the core argument for queues).

Components

1. Notification API / Event Ingestion

Upstream services send notification requests to the notification service API (or directly publish to Kafka):

json

{
  "event_type": "order_placed",
  "user_id": 42,
  "template": "order_confirmation",
  "data": { "order_id": 7821, "total": "$49.99" },
  "channels": ["email", "push"]
}

The API validates the request and publishes to Kafka topic notification-requests.

2. Notification Router / Preference Engine

A consumer reads from notification-requests, looks up:

User's notification preferences (opted out of SMS? push disabled?)
User's device tokens (for push — stored when the app was installed)
User's email address, phone number

It then routes the notification to per-channel Kafka topics: notification-email, notification-push, notification-sms.

Consumer groups allow this routing step to be parallelized without each partition being processed by multiple workers (Q45).

3. Per-channel workers

Separate worker pools consume from each channel topic:

Email workers: render the template (HTML/text from the template field + data), call SendGrid or Amazon SES API, record delivery result
Push workers: construct the payload, call APNs (for iOS) or FCM (for Android) with the device token
SMS workers: call Twilio or AWS SNS with the phone number and message body

Each channel has different throughput characteristics and failure modes — separating them means an SMS outage doesn't affect email delivery, and you can scale push workers independently of email workers.

4. Delivery tracking store

Every notification attempt is logged to a database:

sql

CREATE TABLE notification_log (
  id          BIGINT PRIMARY KEY,
  user_id     BIGINT,
  channel     TEXT,          -- 'email', 'push', 'sms'
  template    TEXT,
  status      TEXT,          -- 'pending', 'sent', 'delivered', 'failed'
  sent_at     TIMESTAMPTZ,
  provider_id TEXT,          -- provider's message ID for tracking
  error       TEXT
);

This log enables: auditing ("was this order confirmation sent?"), debugging, retry tracking, and delivery analytics.

Reliability — at-least-once delivery with idempotency

The notification system must guarantee at-least-once delivery (Q43) — a missed notification is worse than a duplicate. If the push worker crashes after calling FCM but before committing the Kafka offset, the message will be redelivered and FCM will be called again.

To prevent duplicate notifications to users: each notification event carries an idempotency key (event_type + user_id + order_id or a UUID from the upstream service). Before calling the provider, check the notification_log for this idempotency key. If already sent, skip. This makes workers idempotent — safe for at-least-once delivery.

Handling provider failures and retries

Providers (FCM, SendGrid, Twilio) have rate limits, temporary outages, and error codes that require different responses:

Transient errors (5xx, timeouts): retry with exponential backoff + jitter (Q29). Use a dead-letter queue for messages that fail after N retries — inspect manually.

Permanent errors:

Invalid device token (push): the user uninstalled the app. Remove the device token from the user's profile. Don't retry.
Bounced email (email): the email address is invalid. Mark it as invalid in the user profile. Don't retry.
Invalid phone number (SMS): mark and don't retry.

These permanent failures must be handled gracefully — retrying an invalid device token wastes quota and can get your app flagged by APNs.

Device token management (push notifications)

Push notifications require a device token — a string that APNs or FCM issues to the app when it's installed on a device. The app sends this token to your server, which stores it.

sql

CREATE TABLE device_tokens (
  id         BIGINT PRIMARY KEY,
  user_id    BIGINT NOT NULL,
  token      TEXT NOT NULL UNIQUE,
  platform   TEXT NOT NULL,    -- 'ios', 'android'
  active     BOOLEAN DEFAULT TRUE,
  created_at TIMESTAMPTZ
);

A user may have multiple devices (phone + tablet). A push notification for user 42 sends to all active device tokens for that user.

APNs and FCM return feedback when a token is invalid (app uninstalled). Process this feedback and mark tokens inactive — critical for keeping your token store clean and your push delivery rate healthy.

Scaling considerations

Fan-out: a system message ("we're adding a new feature") might need to notify 50 million users. This cannot be a synchronous broadcast — it must be batched and queued over minutes or hours. A bulk notification job creates 50M events and publishes them to Kafka in batches. The consumer pool drains at its sustainable rate.

Priority lanes: transactional notifications (order confirmed, payment failed) should be processed before marketing notifications (weekly digest). Separate Kafka topics or priority queues per urgency level ensure critical notifications aren't delayed behind bulk marketing sends.

User preference lookups: every notification requires a database lookup for user preferences. At 10M notifications/day = 115/sec, this is easily handled with a cache (Redis) of user preferences. TTL of 5 minutes is fine — preferences rarely change.

The mental model

code

Architecture: event-driven, fully asynchronous

Flow:
  Upstream service → Notification API → Kafka (notification-requests)
  Router consumer: lookup preferences + device tokens → route to per-channel topics
  Channel workers: email-topic → SendGrid; push-topic → APNs/FCM; sms-topic → Twilio
  All results logged to notification_log

Reliability:
  At-least-once delivery (Kafka + worker offset commit after send)
  Idempotency key per notification → check log before calling provider
  Retries: transient errors → exponential backoff; permanent → mark invalid, no retry

Channels are independent:
  Separate workers, separate Kafka topics
  SMS outage doesn't affect email; scale push workers separately

Device tokens:
  Stored per user per platform; user may have many devices
  APNs/FCM feedback loop → mark invalid tokens inactive

Scale:
  Fan-out (50M notifications): batch job → Kafka publish over time → drain at rate
  Priority lanes: transactional vs marketing → separate topics → process critical first
  User preference lookups: Redis cache, 5min TTL

Failure modes:
  Provider down: retry with backoff → DLQ after N failures
  Invalid token/email/phone: permanent → mark invalid, stop retrying
  Worker crash: at-least-once → reprocess → idempotency prevents duplicates

A notification system's complexity comes not from the delivery itself — calling SendGrid is one API call — but from the reliability guarantees (at-least-once without duplicate sends), the fan-out scale (millions of users), and the failure handling taxonomy (transient vs. permanent errors, dead device tokens). Getting those three right is what separates a notification system from a notification service that sometimes works.