Phase 5 · Chapter 7

Notification System: Fan-out at Scale

How a single event becomes millions of delivered messages — and the failure modes at every step

A notification system is deceivingly simple on the surface: something happens, tell the relevant people about it. At small scale, a direct API call and a database write suffice. At scale, the same simple requirement becomes one of the hardest problems in distributed systems — the fan-out problem — where one event must trigger millions of individual deliveries across multiple channels to users with different preferences, on devices with different delivery semantics, with at-most-once, at-least-once, or exactly-once guarantees depending on the notification type. After this chapter you will understand the full notification pipeline from event ingestion to final delivery, why fan-out on write and fan-out on read make fundamentally different tradeoffs, how delivery guarantees work for each channel, how to design preference and deduplication systems that prevent notification fatigue, and where the architecture breaks under load. This system appears in nearly every senior engineering interview because it touches queues, databases, external APIs, real-time delivery, and scaling tradeoffs simultaneously.

Part 1: What a Notification System Must Do

Start from requirements, not components. A complete notification system must:

Trigger on events. A new follower, a comment on your post, a payment processed, a test result ready. Every business event that users care about is a potential notification trigger.

Fan out to recipients. One event → one or many recipients. A direct message notifies one user. A new post from someone with 2 million followers notifies 2 million users. The fan-out ratio ranges from 1:1 to 1:millions.

Deliver through multiple channels. The same notification may need to go through:

• In-app (a badge count and a notification panel inside the web app)
• Push notification (mobile or browser, delivered even when app is closed)
• Email (for users who are not online)
• SMS (for critical alerts)

Respect user preferences. User A wants push but not email. User B wants email but not push. User C has muted a specific type of notification. User D is in "do not disturb" mode.

Handle delivery semantics correctly. A "your payment failed" notification must arrive exactly once — duplicate delivery is confusing and damages trust. A "you have a new follower" notification can tolerate duplicate delivery in rare cases. A "system is down" alert to engineers must arrive at least once even if it means occasional duplicates.

Provide reliability. If the notification worker crashes mid-flight, deliveries in progress should not silently drop. The system must recover and complete them.

Scale to volume. During peak events (a major announcement, a system outage) notification volume spikes instantly. The pipeline must absorb the spike without dropping deliveries or collapsing under queue pressure.

Part 2: The Fan-out Problem

A user with 2 million followers posts a photo. Every follower should receive a notification. This is the fan-out problem: one write (the post) must produce 2 million notification records.

The naive approach is to process this synchronously in the request handler: when the post is created, loop through all 2 million followers, insert notification records, send emails. This is catastrophically wrong:

• A loop over 2 million rows in a single API request takes minutes. The user's "post" API call hangs for minutes before returning.
• The database receives 2 million INSERT operations in rapid succession, saturating write capacity.
• If the process crashes at follower 500,000, the first 500,000 get notified and the rest do not. There is no way to resume.
• Memory usage spikes as you fetch all 2 million follower IDs at once.

The solution is to decouple the notification trigger from the notification delivery using a queue.

The API call returns instantly. The 2 million notifications are processed asynchronously by the worker fleet, in parallel, over the following seconds. The queue provides durability — if a worker crashes, the job remains in the queue and another worker picks it up.

Part 3: Fan-out on Write vs Fan-out on Read

There are two fundamental strategies for when to compute "which notifications does this user have."

Fan-out on write (push model)

When an event occurs, immediately compute all recipients and write a notification record to each recipient's notification inbox. When a user opens the app and loads their notifications, read their pre-computed inbox.

Advantages: Notification reads are instant — a single indexed query per user. No computation at read time. Works well for users who check notifications frequently (the data is always ready).

Disadvantages: The write amplification is enormous. 2 million followers × 1 follow event = 2 million rows inserted. For an active platform where popular users post frequently, the notifications table grows at a rate proportional to (posts per hour) × (average followers). This is terabytes per day at scale. Most of these rows are never read — users who check notifications infrequently accumulate thousands of unread notifications that they dismiss without reading. You paid to write them all.

Fan-out on write is correct for systems where: recipients have small fan-out (most users have <1000 followers), the platform needs low notification latency (notifications appear within seconds of the event), or the read path needs to be extremely fast.

Fan-out on read (pull model)

Do not write notification records at event time. Instead, when a user loads their notifications, compute what they should see by reading from the events that occurred and filtering for relevance.

Advantages: No write amplification. One event = one row, regardless of fan-out. Storage scales with events, not with events × followers.

Disadvantages: Notification loads are expensive — multiple queries, joins, application-level filtering. Under high read traffic, this computation saturates database CPU. Latency is higher. Implementing "unread count" requires expensive aggregation queries.

Fan-out on read is correct for: timelines/feeds (not strict notifications), users with very high follower counts where write amplification is prohibitive, systems where notifications are low-frequency or low-priority.

The hybrid approach (how Twitter/Instagram do it)

Separate users into two tiers:

Regular users (< N followers): fan-out on write. When they post, insert notification records for each follower. Fast, predictable storage cost.

Celebrity users (> N followers): fan-out on read. When they post, write only the post record. When their followers load notifications, query the celebrity's recent posts. The celebrities are a small percentage of users but responsible for most of the write amplification.

The threshold N is tuned based on storage and write capacity. Twitter historically used around 10,000 followers as the cutoff.

Part 4: The Delivery Pipeline

After fan-out determines who gets notified, delivery handles the actual sending. Each channel has different mechanics and different failure modes.

In-app notifications

In-app notifications are entries in a notifications table that the frontend polls or receives via WebSocket/SSE. The simplest channel: write a row, the client reads it.

Unread count: The badge showing "you have 3 unread notifications" is the most frequently read piece of data in a notification system. Every page load, every navigation — the client wants the current unread count. Implementing this as SELECT COUNT(*) FROM notifications WHERE recipient_id = $userId AND is_read = false is catastrophically slow at scale. A full table scan for every page load collapses the database.

The solution is a Redis counter per user:

// When a notification is created:
await redis.incr(`unread:${userId}`);

// When a user reads their notifications:
await redis.set(`unread:${userId}`, 0);

// On every page load (fast, no database):
const unreadCount = await redis.get(`unread:${userId}`) ?? 0;

The counter is incremented on write and zeroed on read. It is a cache — occasionally the Redis counter diverges from the actual database count (Redis crash, race condition). Reconcile by running a background job that periodically recomputes the count from the database and resets the Redis counter. Acceptable staleness for a badge count is seconds.

Real-time delivery via WebSocket/SSE: When a notification is created, rather than waiting for the client to poll, push it immediately. The collaboration server architecture from Chapter 6 applies here: WebSocket connections are maintained to a notification server, which publishes new notifications through Redis Pub/Sub to the correct server instance.

// When notification is created for userId 12345:
await redis.publish(`notifications:${userId}`, JSON.stringify(notification));

// Notification WebSocket server subscribed to user channels:
sub.on('message', (channel, message) => {
  const userId = channel.replace('notifications:', '');
  const ws = userConnections.get(userId);
  if (ws?.readyState === WebSocket.OPEN) {
    ws.send(message);
  }
});

Server-Sent Events (SSE) as an alternative to WebSocket: For one-way notification delivery (server → client only), SSE is simpler than WebSocket. SSE uses a standard HTTP connection that stays open; the server pushes events as they arrive. No WebSocket upgrade, no bidirectional protocol, works through most proxies and firewalls. For notifications (server pushing to client) SSE is often the correct choice; WebSocket is correct when you also need client-to-server messages.

Email notifications

Email is sent via a transactional email provider (SendGrid, Postmark, AWS SES). The notification worker enqueues an email job, and an email worker processes it:

// In notification worker, after creating notification record:
if (userPreferences.emailEnabled && notificationType !== 'cursor_update') {
  await emailQueue.add('send_notification_email', {
    to: user.email,
    template: 'new_follower',
    data: { followerName, followerAvatar, notificationId },
  }, {
    attempts: 3,
    backoff: { type: 'exponential', delay: 5000 },
  });
}

Email is inherently asynchronous and best-effort from the sender's perspective. Delivery receipts (bounce, open, click) arrive as webhooks from the email provider and should be stored to inform future sending decisions (stop sending email to addresses that hard-bounce).

Email digests: Sending one email per notification event creates email fatigue and high sending costs. Digest emails — "you have 15 new notifications since yesterday" — batch notifications and send once. The tradeoff: lower notification latency. The implementation: a scheduled job (BullMQ cron) runs hourly or daily, queries undelivered email notifications grouped by user, sends one digest email per user, marks notifications as email-delivered.

Push notifications (mobile and web browser)

Push notifications are delivered through platform-specific infrastructure:

• iOS: Apple Push Notification Service (APNs)
• Android: Firebase Cloud Messaging (FCM)
• Web browsers: Web Push Protocol (via VAPID keys)

The architecture is identical across platforms: your server sends a notification to the platform's push infrastructure, which delivers it to the device even when the app is in the background or closed.

// Send push via FCM (Firebase Cloud Messaging):
import { initializeApp, cert } from 'firebase-admin/app';
import { getMessaging } from 'firebase-admin/messaging';

const message = {
  token: user.fcmToken,    // device token stored when user installs app
  notification: {
    title: 'New follower',
    body: `${followerName} started following you`,
  },
  data: {
    notificationId: notification.id,
    type: notification.type,
  },
};

try {
  await getMessaging().send(message);
} catch (error) {
  if (error.code === 'messaging/registration-token-not-registered') {
    // Device token is invalid (app uninstalled) — remove it
    await db.query('DELETE FROM device_tokens WHERE token = $1', [user.fcmToken]);
  }
}

Token management: Device tokens expire and become invalid (app uninstalled, app re-installed, OS updated). Sending to invalid tokens produces an error from APNs/FCM. Your worker must handle these errors and remove stale tokens. A table device_tokens(user_id, token, platform, created_at) stores tokens and receives deletions when delivery fails with a token-invalid error.

Web push: Browsers support push notifications via the Web Push Protocol with VAPID (Voluntary Application Server Identification). The user grants permission, the browser registers a push endpoint with its push service (Google, Mozilla, etc.), and your server uses the VAPID keys to send authenticated push requests. The library web-push (npm) handles the protocol.

Part 5: Deduplication — Preventing Double Delivery

A notification job fails partway through delivery (the process crashes after sending the email but before marking it sent). On retry, the job runs again and sends the email again. The user receives two identical emails. This is notification spam and it erodes trust.

The solution is idempotent delivery — designing each delivery step so that running it twice produces the same result as running it once.

Idempotency key per notification per channel:

CREATE TABLE notification_deliveries (
  id                BIGSERIAL PRIMARY KEY,
  notification_id   BIGINT NOT NULL REFERENCES notifications(id),
  channel           VARCHAR(20) NOT NULL,  -- 'email', 'push', 'sms'
  status            VARCHAR(20) NOT NULL DEFAULT 'pending',
  delivered_at      TIMESTAMPTZ,
  idempotency_key   VARCHAR(64) UNIQUE NOT NULL,
  UNIQUE(notification_id, channel)
);

Before sending, insert a delivery record with status 'pending' using the idempotency key as a unique constraint. If the INSERT succeeds, proceed with delivery and update status to 'delivered'. If the INSERT fails (duplicate key), this delivery has already been attempted — skip it. This pattern is safe for retries: the second attempt finds the existing record and exits without re-sending.

async function deliverEmail(notificationId, userId) {
  const key = `email:${notificationId}`;

  // Atomic check-and-insert
  const result = await db.query(`
    INSERT INTO notification_deliveries (notification_id, channel, idempotency_key)
    VALUES ($1, 'email', $2)
    ON CONFLICT (notification_id, channel) DO NOTHING
    RETURNING id
  `, [notificationId, key]);

  if (result.rows.length === 0) {
    // Already delivered or in progress — skip
    return;
  }

  // Proceed with delivery
  await sendEmail(notificationId, userId);

  await db.query(
    'UPDATE notification_deliveries SET status = $1, delivered_at = NOW() WHERE idempotency_key = $2',
    ['delivered', key]
  );
}

Part 6: Notification Preferences and Do Not Disturb

Every user has preferences that gate delivery. A preferences system must support:

Per-channel preferences: Enable/disable email, push, SMS independently. Per-type preferences: Mute "new follower" notifications but keep "payment failed" notifications. Do not disturb: Suppress all non-critical notifications between 11pm and 8am (user's timezone). Muted entities: Mute notifications from a specific user or thread.

CREATE TABLE notification_preferences (
  user_id           BIGINT PRIMARY KEY REFERENCES users(id),
  email_enabled     BOOLEAN DEFAULT true,
  push_enabled      BOOLEAN DEFAULT true,
  sms_enabled       BOOLEAN DEFAULT false,
  dnd_enabled       BOOLEAN DEFAULT false,
  dnd_start         TIME DEFAULT '23:00',
  dnd_end           TIME DEFAULT '08:00',
  timezone          VARCHAR(50) DEFAULT 'UTC',
  updated_at        TIMESTAMPTZ DEFAULT NOW()
);

CREATE TABLE notification_type_preferences (
  user_id           BIGINT REFERENCES users(id),
  notification_type VARCHAR(50),
  channel           VARCHAR(20),
  enabled           BOOLEAN DEFAULT true,
  PRIMARY KEY (user_id, notification_type, channel)
);

Preferences are read on every notification delivery. Cache them in Redis with a short TTL (5 minutes). Preference reads are extremely frequent; a database query per notification per user collapses read capacity.

async function getUserPreferences(userId) {
  const cacheKey = `prefs:${userId}`;
  const cached = await redis.get(cacheKey);
  if (cached) return JSON.parse(cached);

  const prefs = await db.query(
    'SELECT * FROM notification_preferences WHERE user_id = $1',
    [userId]
  );
  await redis.setex(cacheKey, 300, JSON.stringify(prefs.rows[0]));  // cache 5 minutes
  return prefs.rows[0];
}

function shouldDeliver(prefs, notificationType, channel, now) {
  if (!prefs[`${channel}_enabled`]) return false;

  if (prefs.dnd_enabled) {
    const userTime = toUserTimezone(now, prefs.timezone);
    if (isDuringDnd(userTime, prefs.dnd_start, prefs.dnd_end)) {
      // Suppress unless it's a critical notification type
      if (!CRITICAL_NOTIFICATION_TYPES.includes(notificationType)) return false;
    }
  }

  return true;
}

Part 7: The Complete Pipeline

Assembling all the components into one coherent system:

Part 8: Failure Modes and Resilience

Queue backup: During a viral event, the fan-out queue fills faster than workers can drain it. Notifications are delayed by minutes. Fix: auto-scale the worker fleet based on queue depth (a metric you are emitting from Phase 4 Chapter 5). Set a maximum queue depth alert. For truly viral events, implement rate limiting on the fan-out layer — spread notifications over a longer window rather than attempting all 2 million immediately.

Email provider rate limits: Transactional email providers have sending limits. At 10 million emails per hour, you exceed most providers' limits. Fix: use multiple email providers (primary

• fallback), implement sending rate limiting in the email worker, and negotiate higher limits with your provider before hitting them. Monitor bounce rates and sending reputation — a damaged sender reputation causes all your emails to land in spam.

Push token churn: Large-scale user base, lots of app reinstalls, token invalidation. Sending to expired tokens wastes FCM/APNs API calls and fills logs with errors. Fix: run a weekly cleanup job that retries delivery to tokens that have been failing for 7+ days with token-invalid errors and removes them.

Notification storms on recovery: Your notification system goes down for 2 hours. When it recovers, it has 50 million queued notifications to deliver. Users who have been offline for 2 hours receive 47 push notifications at once. This is worse than missing the notifications. Fix: implement notification coalescing — if more than N notifications of the same type are queued for a user, collapse them into one summary notification. "You have 47 new followers" is better than 47 individual push notifications.

Vocabulary Map

Practice Problems

Problem 1 — Fan-out Strategy: AttemptIQ launches a "follow teachers" feature. Teachers post study materials and students get notified. 90% of teachers have fewer than 500 students. 2% of teachers (the most popular ones) have 50,000–500,000 students.

Design the fan-out strategy. Which users use fan-out on write? Which use fan-out on read? What is the threshold and why? For a top teacher with 400,000 students who posts 10 times per day: calculate the daily notification row count under fan-out on write. Calculate the database read cost per student page load under fan-out on read. Which is correct for this use case?

Problem 2 — Unread Count at Scale: Samiksha has 200,000 active users. Each user loads their notification panel 8 times per day on average. The notification panel shows: unread count badge, last 20 notifications, and a "mark all read" button.

Design the data model and query plan for all three features. Why is SELECT COUNT(*) FROM notifications WHERE recipient_id = $userId AND is_read = false wrong at this scale? Write the Redis-based unread count design with the exact INCR and SET operations. What is the edge case where the Redis counter diverges from reality and how do you detect and fix it?

Problem 3 — Delivery Guarantee Design: Classify the following notification types by required delivery guarantee (at-most-once, at-least-once, exactly-once) and justify each:

a) "Your test result is ready" (AttemptIQ) b) "New follower" (social feature) c) "Payment of ₹499 was charged" (Samiksha billing) d) "System maintenance in 30 minutes" (broadcast to all users) e) "Your answer evaluation failed, please retry" (Samiksha)

For each, describe what the failure cost is in each direction: what happens if it is delivered twice, and what happens if it is not delivered at all? How does this inform your choice?

Problem 4 — Go Deeper: Your notification system sends email via SendGrid. SendGrid delivers the email to the user's inbox. The user's email client downloads the email. Your system recorded status = 'delivered' when SendGrid acknowledged the send.

There are four distinct meanings of "delivered" in this chain. List them in order. Which one does your notification_deliveries table actually record? What additional data would you need from SendGrid webhooks to track further down the chain? What privacy implications arise from tracking email opens and what does this mean for your EU users under GDPR?

Problem 5 — Connect to Previous Phases: In Phase 3 Chapter 4, you studied Kafka as an append-only log for event streaming. In Phase 4 Chapter 4, you used BullMQ for job queues. In this chapter, the notification pipeline uses both.

Explain specifically why each technology is used at its layer: why is Kafka correct for the ingestion layer (business events → fan-out) but BullMQ correct for the delivery layer (fan-out → email/push/in-app)? What property of Kafka makes it correct for event ingestion that BullMQ lacks? What property of BullMQ makes it correct for delivery jobs that Kafka makes harder? If you removed Kafka and used BullMQ end-to-end, what specific capability would you lose? If you removed BullMQ and used Kafka end-to-end, what would become more complex?

What's Next

This chapter completes Phase 5 and the curriculum. You have now designed seven production systems end to end — URL shortener, Twitter feed, YouTube, WhatsApp, Uber — including two of the hardest problems in the field: real-time collaborative editing and large-scale notification delivery. Every design decision in these case studies connects back to the foundations you built in Phases 1 through 4: the CRDT's convergence property is CAP theorem applied to a data structure, the notification pipeline's fan-out is the same problem as Twitter's timeline at a different layer, the unread count Redis pattern is the same pattern as session storage and rate limiting.

The curriculum is now complete. What remains is deliberate practice: taking any system you can think of, designing it from scratch, and asking yourself at every component — what breaks here, what scales here, and what tradeoff am I making? The systems in Phase 5 are models for that practice. Use them as templates, not answers.

End of Phase 5, Chapter 7. End of the Backend & Distributed Systems Curriculum.