Q72 — Design a webhook delivery system

What the system must do

A webhook is an HTTP callback: when an event occurs in System A (a payment completes, an order ships, a user signs up), System A makes an HTTP POST to a URL that System B registered — delivering the event payload in real time.

Your system is like Stripe or GitHub: other companies register webhook endpoints with you, and when events occur on your platform, you reliably deliver HTTP POST requests to their servers.

Requirements:

• Customers register webhook endpoints (URL + event types to receive)
• When a matching event occurs, deliver a POST request to the registered URL
• Reliable delivery: if the endpoint is temporarily down, retry with backoff
• At-least-once delivery (the receiver must be idempotent)
• Delivery status visibility (did this webhook succeed?)
• Scale: millions of events per day, thousands of registered endpoints

Why webhooks are hard — you're calling systems you don't control

Sending an HTTP request is easy. Sending it reliably to a system you don't own, that may be down, slow, returning errors, or mishandling your requests — that's the challenge.

Customer endpoints:

• Go down for maintenance
• Return 500 errors under load
• Respond slowly (30+ second timeouts)
• Return 200 OK but silently discard the payload
• Change URLs without telling you

You must handle all of these gracefully, retrying when appropriate, giving up when appropriate, and giving customers visibility into what happened.

Components

1. Event ingestion

When an event occurs (payment completed), it's published to Kafka topic events. Every event has a type, payload, and timestamp. The webhook delivery system subscribes to this topic.

2. Fan-out / subscription matching

A consumer reads from events and looks up: which webhook subscriptions match this event type? A subscription is:

CREATE TABLE webhook_subscriptions (
  id         BIGINT PRIMARY KEY,
  customer_id BIGINT NOT NULL,
  url        TEXT NOT NULL,
  events     TEXT[] NOT NULL,  -- e.g., ['payment.completed', 'order.shipped']
  secret     TEXT NOT NULL,    -- for request signing
  active     BOOLEAN DEFAULT TRUE
);

For each matching subscription, create a delivery job and enqueue it.

3. Delivery queue

Delivery jobs go into a Kafka topic webhook-deliveries or a database-backed job queue (Q67). Each job has: subscription_id, endpoint URL, event payload, attempt_count, next_attempt_at.

A pool of delivery workers consumes from this queue, makes the HTTP POST, and records the result.

4. Delivery workers

For each job:

1. Build the HTTP request: POST to the endpoint URL, body = event JSON, headers include a signature
2. Set a reasonable timeout (e.g., 10 seconds — customer endpoints must respond quickly)
3. Send the request
4. Record the result: status code, response body, latency

If the response is 2xx → mark delivered. If 4xx (bad request, auth failure) → don't retry (permanent failure). If 5xx, timeout, or network error → retry with exponential backoff.

Retry strategy

The retry schedule must be respectful of the customer's server while ensuring eventual delivery:

Attempt 1: immediately
Attempt 2: 5 minutes later
Attempt 3: 30 minutes later
Attempt 4: 2 hours later
Attempt 5: 6 hours later
Attempt 6: 24 hours later
→ After 6 attempts or 3 days: mark as permanently failed, alert customer

Add jitter (Q29) to each delay to prevent all retries from a traffic spike hitting the customer's server simultaneously.

Retries are implemented by updating the job's next_attempt_at in the queue and incrementing attempt_count. The job scheduler (Q67) picks it up when next_attempt_at arrives.

Webhook signatures — proving authenticity

When a customer's server receives a POST, how do they know it came from you (and not from an attacker spoofing events)?

HMAC signing: when the customer registers their endpoint, you generate a shared secret. For each delivery, compute:

signature = HMAC-SHA256(secret, timestamp + "." + raw_payload_body)

Include this in the request header: X-Webhook-Signature: sha256=<hex>. Include the timestamp: X-Webhook-Timestamp: 1710000000.

The customer verifies: compute the same HMAC with their secret and compare. They also check the timestamp is recent (within 5 minutes) to prevent replay attacks — an attacker capturing the signed request and replaying it later.

This is how Stripe, GitHub, and Twilio authenticate their webhooks.

Delivery log and customer visibility

Customers need to see: did my webhook fire? Did it succeed? What did your server return?

CREATE TABLE webhook_delivery_log (
  id                BIGINT PRIMARY KEY,
  subscription_id   BIGINT NOT NULL,
  event_id          BIGINT NOT NULL,
  attempt_number    INT NOT NULL,
  url               TEXT NOT NULL,
  request_body      JSONB,
  response_status   INT,
  response_body     TEXT,
  latency_ms        INT,
  status            TEXT,  -- 'success', 'failed', 'retrying'
  delivered_at      TIMESTAMPTZ,
  next_retry_at     TIMESTAMPTZ
);

This log is the foundation of the customer-facing webhook dashboard: "Here are all delivery attempts for this event. Attempt 1 failed (503). Attempt 2 succeeded (200 OK). Here's the response body."

Customers can also manually trigger a redelivery from the dashboard — useful for debugging.

Handling slow endpoints — timeout enforcement

A customer endpoint that takes 30 seconds to respond ties up a worker for 30 seconds. At scale, this can exhaust your worker pool if many customers have slow endpoints.

Timeouts are mandatory: every HTTP call must have a strict timeout (e.g., 10 seconds). After 10 seconds with no response, abort and treat as failure (retry later).

Bulkhead pattern: don't share a worker pool across all customers. If one customer's slow endpoint fills the pool, other customers' deliveries are delayed. Use per-customer queues or worker pools with isolation — a slow customer only hurts themselves, not others.

Circuit breaker per endpoint

If a customer's endpoint consistently fails (5xx on every attempt for 10 minutes), continuing to hammer it is wasteful. Implement a circuit breaker (Q31) per endpoint:

• After N consecutive failures: open the circuit → stop sending (except test requests every 5 minutes)
• When the test request succeeds: close the circuit → resume normal delivery
• While circuit is open: queue incoming events but don't attempt delivery (they'll be sent once circuit closes)

Alert the customer when their circuit opens: "Your webhook endpoint has been failing for 10 minutes. Please check it."

The mental model

Flow:
  Event → Kafka → subscription matcher → delivery jobs → delivery workers → HTTP POST to customer

Delivery workers:
  POST with 10s timeout
  2xx → success; 4xx → permanent fail (no retry); 5xx/timeout → retry

Retry:
  Exponential backoff with jitter: 5m, 30m, 2h, 6h, 24h
  Max 6 attempts over 3 days → permanently failed + customer alert

Authenticity:
  HMAC-SHA256(secret, timestamp + payload) in X-Webhook-Signature header
  Customer verifies signature + timestamp freshness (replay prevention)

Delivery log:
  Every attempt logged: status, response, latency
  Customer dashboard: inspect attempts, trigger manual redelivery

Reliability:
  Timeout enforcement (10s per call)
  Bulkhead isolation per customer (slow customer ≠ affecting others)
  Circuit breaker: stop hammering consistently-failing endpoints

Scale:
  Millions of events/day: Kafka handles ingestion burst
  Thousands of endpoints: worker pool scaled horizontally
  Delivery jobs in DB queue or Kafka with next_attempt_at scheduling

The core guarantee:
  At-least-once delivery (retry on failure → idempotency required at receiver)
  Receiver should check event_id before processing to handle duplicates

A webhook delivery system is a reliability and trust problem disguised as an HTTP client. The HTTP POST is one line of code; everything else — retries, backoff, signatures, circuit breakers, delivery logs, bulkhead isolation, timeout enforcement — exists to make that one line of code reliable when the endpoint on the other side is outside your control.

End of Question Bank

72 questions. 12 parts. Every foundational concept in distributed systems, from TCP to collaborative editing, built from first principles with no gaps.