Chapter 23: Design a Payment Processing System

Curriculum: Distributed Systems from First Principles Chapter: 23 of 24 Part: 10 — Synthesis Design Problems Framework: Chapter 19's six-step process, applied in full Chapters exercised: 2 (WAL, durability), 3 (indexes — the audit trail), 4–6 (replication, consistency — correctness over availability), 11–13 (consensus, Raft, distributed transactions), 14–15 (idempotency, Kafka for async pipelines), 17 (rate limiting), 18 (observability — the audit log), 19 (design framework) Estimated reading time: 85–95 minutes

The Problem Statement

Design a payment processing system like Stripe: businesses integrate your API to charge customers' credit and debit cards, transfer money between bank accounts, and issue refunds. The system must process 1 million payment attempts per day, ensure that money is never created or destroyed by any failure scenario, be auditable (every dollar has a traceable history), and respond to payment requests within 3 seconds. Design for Stripe's scale.

Opening: Why Payments Are Different

Every system in this curriculum involves trade-offs between correctness and performance. Redis chooses performance over durability. Cassandra chooses availability over strong consistency. The CDN chooses stale-but-fast over fresh-but-slow. In every case, the failure mode is recoverable: stale data refreshes, an inconsistent cache repairs itself, a dropped message is retried.

In a payment system, the failure modes are not recoverable:

• Double charge: The customer is billed twice for one purchase. Money is created from nothing in the merchant's account. The customer is furious. Reversing it requires human intervention, refund processing, and potentially regulatory reporting.
• Lost payment: The customer's card is charged but the merchant's balance is not credited. Money disappears. The merchant does not fulfill the order. The customer disputes the charge. Discovering this requires reconciliation against bank statements, days later.
• Phantom refund: A refund is processed twice. The merchant loses money they were never meant to lose. The bookkeeping is wrong until the next reconciliation run.

None of these are "briefly inconsistent" scenarios that self-heal. They are ledger errors — permanent, visible, and expensive to correct. Regulatory auditors, bank partners, and angry customers all have claims on your attention.

This changes the design philosophy entirely. In a payment system: correctness is non-negotiable, and you trade availability and performance to achieve it. A payment that takes 5 seconds is acceptable. A payment that charges the customer twice is not. If you must choose between "the payment fails cleanly" and "the payment might succeed ambiguously," always choose clean failure. An ambiguous success is a potential double charge waiting to be discovered.

This chapter is the culmination of the curriculum's distributed systems concepts — WAL, ACID transactions, consensus, idempotency, distributed transactions — applied to the domain where their correctness guarantees matter most.

Step 1: Requirements Clarification

Functional Requirements

1. Charge a card: A merchant provides a card number (or tokenized card reference), amount, and currency. The system contacts the card network (Visa/Mastercard), authorizes the charge, captures the funds, and credits the merchant's balance.
2. Refund a payment: A merchant can refund a previous charge, in full or partially. The funds are returned to the cardholder and debited from the merchant's balance.
3. Pay out to merchant: Accumulated merchant balances are transferred to their bank account (ACH/wire) on a rolling schedule.
4. Transfer between accounts: Move money between two accounts within the platform (internal transfers for marketplace payments — e.g., Uber paying drivers).
5. Retrieve payment status: Merchants can query the status of any payment at any time.
6. Webhooks: The system notifies merchants of payment status changes (succeeded, failed, refunded) via HTTP callbacks.

Scope out: multi-party escrow, cryptocurrency, complex financial instruments. Focus on card payments and bank transfers.

Non-Functional Requirements

• Scale: 1M payment attempts/day = ~11.6 attempts/second average. Peak (10× average): ~120/second. (Payments are lower volume than messaging or CDN but far higher value per operation.)
• Latency: Payment response within 3 seconds (card network round-trip dominates — typically 1–2 seconds). Internal processing must add < 500ms.
• Correctness: Zero tolerance for double charges, lost payments, or phantom refunds. The ledger must always balance: sum of all debits = sum of all credits.
• Idempotency: Every operation must be idempotent. Retrying a payment request must not create a second charge.
• Availability: 99.99%. Payment downtime directly costs merchants revenue. But: if availability and correctness conflict, choose correctness (fail the payment cleanly).
• Durability: Once a payment is confirmed as succeeded, it must survive any failure — disk failure, node crash, datacenter outage.
• Auditability: Every state change in every payment must be logged with timestamp, actor, and reason. The audit trail is a legal requirement.
• Security: Card data (PAN — Primary Account Number) must never be stored in plaintext. PCI DSS compliance is mandatory.

Step 2: Estimate Scale

Payment volume:
  1M attempts/day / 86,400s = 11.6 attempts/second average
  Peak: 120 attempts/second (holiday shopping, end-of-month billing)

Database write load:
  Each payment attempt generates ~10 database writes:
    - Payment record insert
    - 3-5 state transition updates (PENDING → PROCESSING → SUCCEEDED)
    - 2-3 ledger entries (debit customer, credit merchant, fee)
    - 1-2 event records (for webhooks)
  Peak: 120 × 10 = 1,200 writes/second → trivial for PostgreSQL

Financial volume:
  Average transaction: $75
  1M transactions/day × $75 = $75M/day processed
  Annual: ~$27B (Stripe processes ~$1T/year — we're at ~3% of Stripe scale)

Ledger storage:
  Each ledger entry: ~500 bytes (amount, currency, from_account, to_account, timestamp, payment_id, description)
  1M payments/day × 4 ledger entries avg × 500 bytes = 2 GB/day
  10-year retention (regulatory requirement): 7.3 TB → single PostgreSQL instance

Audit log storage:
  Each payment event: ~1 KB (full state snapshot + metadata)
  1M payments × 10 events × 1 KB = 10 GB/day
  10-year retention: 36.5 TB → needs partitioned storage or cold archival

Card tokenization:
  1M unique customers × card metadata (no PAN): ~200 bytes each = 200 MB
  Trivially small — the cardholder data environment (CDE) is isolated.

The numbers reveal something important: a payment system is not a high-volume distributed systems problem. At 120 writes/second peak, a single PostgreSQL instance handles this easily. The hard problem is not scale — it's correctness. The architecture is designed around preventing data anomalies, not around horizontal scaling.

This is the opposite of the CDN (where scale was the hard problem) and the messaging system (where fan-out was the hard problem). Payment processing is hard because of the correctness requirements, the multi-party coordination (your system + card network + bank), and the consequences of getting it wrong.

Step 3: The Ledger — Double-Entry Bookkeeping

Every financial system since 1494 (when Luca Pacioli codified it) uses double-entry bookkeeping: every financial event produces at least two ledger entries that sum to zero. Money is not created or destroyed — it moves from one account to another.

DOUBLE-ENTRY RULE:
  Every transaction: sum of all debits = sum of all credits
  Equivalently: sum of all ledger entries = 0 (credits are positive, debits negative)

Example: Customer pays $100 to Merchant A, Stripe takes $2.90 fee.

Ledger entries:
  account_id=customer_card,     amount=-100.00, currency=USD  (debit customer)
  account_id=merchant_A,        amount=+97.10,  currency=USD  (credit merchant)
  account_id=stripe_revenue,    amount=+2.90,   currency=USD  (credit fee account)

Check: -100.00 + 97.10 + 2.90 = 0.00 ✓ Balanced.

Example: Refund the $97.10 to customer (Stripe keeps the $2.90 fee):
  account_id=merchant_A,        amount=-97.10,  currency=USD  (debit merchant)
  account_id=customer_card,     amount=+97.10,  currency=USD  (credit customer)

Check: -97.10 + 97.10 = 0.00 ✓ Balanced.

The ledger is the ground truth. Every balance you show a merchant is derived by summing their ledger entries: SELECT SUM(amount) FROM ledger WHERE account_id = 'merchant_A'. If the ledger sum is wrong, the balance is wrong. Balances are not stored separately — they're always computed from the ledger. This eliminates the entire class of bugs where "the balance field" and "the ledger" get out of sync (Chapter 2's consistency problem between two pieces of state).

The Ledger Table

CREATE TABLE ledger_entries (
    entry_id        BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    payment_id      UUID NOT NULL,           -- links to the payment
    account_id      VARCHAR(64) NOT NULL,    -- which account (merchant, customer, fee)
    amount          NUMERIC(19,4) NOT NULL,  -- exact decimal, never float
    currency        CHAR(3) NOT NULL,        -- ISO 4217: USD, EUR, GBP
    entry_type      VARCHAR(32) NOT NULL,    -- 'PAYMENT', 'REFUND', 'FEE', 'PAYOUT'
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    idempotency_key VARCHAR(255) NOT NULL,   -- prevents duplicate entries
    metadata        JSONB,                   -- audit context

    CONSTRAINT ledger_entries_idempotency UNIQUE (idempotency_key)
);

CREATE INDEX idx_ledger_account ON ledger_entries (account_id, created_at);
CREATE INDEX idx_ledger_payment ON ledger_entries (payment_id);

Why NUMERIC(19,4) and never FLOAT? Floating-point arithmetic is imprecise. 0.1 + 0.2 = 0.30000000000000004 in IEEE 754 floating-point. In a financial ledger, this imprecision accumulates. NUMERIC (PostgreSQL's arbitrary-precision decimal) is exact: 0.1 + 0.2 = 0.3 exactly. A system that uses FLOAT for money values will have a ledger that is perpetually slightly unbalanced — discovered during reconciliation, never perfectly fixable. Always use NUMERIC/DECIMAL for money.

The ledger is append-only. Entries are never updated or deleted. Corrections are made by adding new entries that reverse the incorrect ones. This is why the audit trail is complete — every financial event is a permanent record. This is also Chapter 10's "LSM insight" applied to financial data: sequential writes are optimal, and immutability simplifies both correctness reasoning and auditing.

Step 4: Idempotency — The Most Important Concept in Payments

Here is the fundamental challenge: a merchant's server calls your payment API. The payment request travels across the network, your system processes it, the card is charged, but the response is lost in transit (network failure). The merchant's server times out waiting. Should it retry?

If the merchant retries without idempotency: Two identical payment requests arrive. If your system cannot distinguish a retry from a new payment, the card is charged twice. Double charge.

If the merchant does not retry: The payment may or may not have succeeded (it did — but the merchant doesn't know). The order is not fulfilled. The customer is charged but receives nothing.

Both outcomes are unacceptable. The solution: idempotency keys.

Idempotency Keys

The merchant generates a unique key for each distinct payment attempt (typically a UUID). They include it in every API request. If they retry (due to network failure), they use the same key. Your system uses the idempotency key to detect retries:

The merchant receives the same response both times. No double charge. The idempotency key transforms a non-idempotent operation (charging a card) into a safe-to-retry one.

Idempotency Key Storage and Locking

The check-and-set operation ("has this key been used? if not, mark it as used") must be atomic. If two retries arrive simultaneously, both might read "not used" and both start processing — double charge again.

-- Idempotency keys table
CREATE TABLE idempotency_keys (
    idempotency_key VARCHAR(255) PRIMARY KEY,
    request_hash    BYTEA NOT NULL,       -- hash of request body (detect mismatched retries)
    response        JSONB,                -- cached response (NULL if still processing)
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    locked_at       TIMESTAMPTZ,          -- when processing started
    completed_at    TIMESTAMPTZ           -- when processing finished
);

-- Atomic lock acquisition: the PRIMARY KEY makes "claim this key" a single atomic step.
INSERT INTO idempotency_keys (idempotency_key, request_hash, locked_at)
  VALUES ('idem_3f8c1d2e4b6789a0', $hash, NOW())
  ON CONFLICT (idempotency_key) DO NOTHING
  RETURNING idempotency_key;
-- 1 row returned  → we inserted it: this is the first request, proceed with processing
-- 0 rows returned → the key already existed; read it:
SELECT request_hash, response, locked_at
  FROM idempotency_keys
 WHERE idempotency_key = 'idem_3f8c1d2e4b6789a0';
-- request_hash differs   → 422: same key reused for a different request
-- response IS NOT NULL   → return the cached response immediately
-- response IS NULL       → another request is processing → return 409 or wait

Why DO NOTHING and not DO UPDATE SET locked_at = NOW(). A tempting version uses ON CONFLICT ... DO UPDATE SET locked_at = NOW() RETURNING *, with a comment that the update is a "no-op if the row exists." It isn't: it overwrites locked_at on every retry, and RETURNING * then returns a row in both cases, so the code can no longer tell "I just claimed this key" from "someone else already holds it." Two concurrent retries would both proceed and charge the card twice. It also resets the lock timestamp on every retry, so you can never detect a stale lock — a key whose first request crashed mid-processing (response IS NULL and locked_at older than, say, 60 seconds). Such a key must be recovered (check the payment's state, then finish or release it), not left blocking every retry with 409 forever. DO NOTHING + RETURNING keeps the signal clean: a returned row means you own the key.

The "in-flight" problem: What if the first request is still processing when the retry arrives? The idempotency key exists (locked) but has no response yet. Options:

• Return HTTP 409 Conflict with a "request in flight, retry later" message
• Block the retry for a short timeout, return result when first request completes
• Return a "processing" status, let merchant poll for completion

Stripe returns HTTP 409 and instructs the merchant to retry after the first request completes. This is the clean approach — it avoids complex waiting logic and makes the system stateless per request.

Mismatched retries: A merchant sends retry with the same idempotency key but different request body (amount: 2000 instead of 1000). The request hash doesn't match. Return HTTP 422 Unprocessable Entity: "idempotency key already used with different parameters." This prevents a subtle attack: using an idempotency key from a small payment to authorize a large one.

Step 5: The Payment State Machine

A payment is not a single atomic operation. It is a sequence of states, each representing a distinct step in the process. Modeling this explicitly as a state machine is essential for correctness.

Why model it as a state machine? Three reasons:

First, each state is a durable checkpoint. If a payment crashes between authorization and capture, the system can restart and know exactly where to resume. "We authorized but didn't capture — should we capture or void the auth?" The state machine answers this.

Second, invalid transitions are rejected. A refund on a DECLINED payment is rejected. A double capture is rejected. Explicit modeling prevents nonsensical operations.

Third, the state history is the audit trail. Every state transition is logged with timestamp and reason. Regulators can ask "why was this payment DECLINED?" and the answer is in the state history.

Payment Table

CREATE TABLE payments (
    payment_id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    idempotency_key     VARCHAR(255) UNIQUE NOT NULL,
    merchant_id         VARCHAR(64) NOT NULL,
    customer_id         VARCHAR(64) NOT NULL,
    amount              NUMERIC(19,4) NOT NULL,
    currency            CHAR(3) NOT NULL,
    status              VARCHAR(32) NOT NULL DEFAULT 'PENDING',
    card_token          VARCHAR(255) NOT NULL,  -- never store raw PAN
    card_network_auth_id VARCHAR(255),          -- Visa/MC authorization code
    failure_code        VARCHAR(64),            -- 'CARD_DECLINED', 'INSUFFICIENT_FUNDS', etc.
    failure_message     VARCHAR(512),
    created_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    updated_at          TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    metadata            JSONB
);

CREATE TABLE payment_events (
    event_id        BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    payment_id      UUID NOT NULL REFERENCES payments(payment_id),
    from_status     VARCHAR(32),
    to_status       VARCHAR(32) NOT NULL,
    actor           VARCHAR(64) NOT NULL,   -- 'system', 'merchant', 'card_network'
    reason          TEXT,
    metadata        JSONB,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
-- payment_events is append-only — never UPDATE or DELETE

Every state transition writes a row to payment_events. This is Chapter 2's WAL concept applied at the application level: before changing the payment's status, write the transition record first. If the system crashes after writing the event but before updating the payment status, the event record allows recovery.

Step 6: Card Network Integration — Authorization vs. Capture

Credit card payments involve two distinct operations that many engineers conflate:

Authorization (auth): The card network (Visa, Mastercard) verifies that the card is valid, the account has sufficient credit, and the merchant is allowed to charge this card. The cardholder's credit limit is "held" (reduced by the authorized amount). The merchant has not yet received money. Authorization codes are valid for 7 days (Visa) or 30 days (Mastercard).

Capture: The merchant requests that the authorized funds actually move from the cardholder's account to the merchant's acquiring bank. This is the actual money movement. Most card authorizations are captured within 24 hours.

Why separate auth and capture? Two reasons. First, merchants often don't know the final amount at authorization time (e.g., hotel holds $200, actual bill is $187 — they capture for $187). Second, if the order cannot be fulfilled (item out of stock), the merchant can void the authorization instead of refunding — the cardholder's credit is released immediately without waiting for a refund cycle.

Pre-authorization (hotel/gas station pattern): A gas station doesn't know how much fuel you'll pump. They authorize $100 (the max charge), wait until you're done, then capture the actual amount (e.g., $43.17). The remaining $56.83 authorization is voided. Your system must handle partial captures and auth-void operations.

Auth without capture (zero-dollar auth): Used to verify card validity without charging. The cardholder's statement shows a $0 authorization. Used by subscription services when adding a new card.

Step 7: The Charge Flow — Correctness Under Failures

The most dangerous part of the payment system is the call to the card network. It is a synchronous HTTP call to an external system over which you have no control. It can fail, time out, or return ambiguous responses. The correctness of the entire system depends on handling these cases correctly.

The timeout case is the hardest. A 30-second timeout waiting for Visa means: Visa received the request (or didn't), processed it (or didn't), and the response was lost. You cannot safely assume either outcome. The recovery pattern — query Visa about the specific payment ID — is exactly how Stripe handles this. Visa's API supports "retrieve authorization by idempotency key," and all major card networks expose this endpoint for exactly this purpose.

Passing your payment_id to the card network: Always send your internal payment_id (or a derived reference) to the card network in every authorization request. This is the shared correlation key that enables the recovery pattern. Without it, you cannot query the card network to resolve timeouts.

Step 8: Distributed Money Movement — The Saga Pattern

A marketplace payment is a distributed transaction across multiple accounts:

Customer pays $100 for a service.
Marketplace platform:
  - Collects $100 from customer's card
  - Credits $90 to the service provider's account
  - Keeps $10 as platform fee

These are three separate operations on three separate accounts.
If any step fails after partial completion, money is stranded.

Chapter 13's 2PC is tempting here — coordinate all three operations atomically. But 2PC requires a coordinator that can block if it crashes (Chapter 13's blocking problem), and one of the participants is an external card network that doesn't speak 2PC. 2PC cannot cross company boundaries.

The correct approach: the Saga pattern (Chapter 13). Model the payment as a sequence of local transactions, each with a compensating action:

Critical: each compensating action must be idempotent. If the compensation itself fails and is retried, it must not double-compensate.

Saga orchestration (Chapter 13's orchestrator model): A centralized saga orchestrator service manages the payment saga state:

PaymentSaga {
  saga_id: UUID
  payment_id: UUID
  current_step: INT       (1, 2, 3, 4, 5)
  direction: ENUM         (FORWARD, COMPENSATING)
  step_states: JSONB      (outcome of each completed step)
  status: ENUM            (RUNNING, SUCCEEDED, COMPENSATED, STUCK)
}

The orchestrator is durable — it writes to PostgreSQL before calling any external service. If it crashes at any point, it restarts and reads its current state from the database, resuming exactly where it left off. This is the saga's equivalent of Chapter 12's Raft log — the durable state record that enables crash recovery.

Stuck sagas: A saga gets stuck when a compensating action fails and cannot be retried automatically. Example: refund fails because the card has been reported stolen and canceled. A human must intervene. Stuck sagas must be monitored (Chapter 18's alerting: payment_sagas_stuck gauge > 0 triggers immediate page) and resolved via an internal operations dashboard. Every payment system has a small percentage of stuck sagas that require manual resolution.

Step 9: The Payout — Money Leaving the Platform

Merchant balances accumulate in the platform's internal ledger. The payout moves real money from the platform's bank account to the merchant's bank account via ACH (Automated Clearing House, the US bank transfer network) or wire transfer.

The 1–2 day settlement window is not a technical delay — it is how ACH works. Same-day ACH (introduced 2016) reduces this to same-day but costs more. Wire transfers settle immediately but cost $15–25 per transfer (only used for large amounts).

The payout ledger trick: Notice that the ledger entries are written atomically with the payout record — the merchant's balance is debited before the money actually leaves the bank. This is intentional. If the merchant requests a payout and then immediately requests another payout before the first ACH settles, their balance correctly shows $0 available (not $500 they've already requested). The ledger entry acts as a "reservation" — the money is logically moved even though physical movement is pending.

Step 10: Card Tokenization — PCI DSS Compliance

Raw card numbers (PANs — Primary Account Numbers) are the most sensitive data in existence. PCI DSS (Payment Card Industry Data Security Standard) mandates that systems that store, transmit, or process PANs comply with 300+ security requirements. The easiest way to comply: don't store PANs.

Tokenization:

The main payment system, the merchant API, the ledger database — none of these ever see the raw PAN. Only the tokenization service and the card network gateway touch card numbers. This isolation dramatically reduces PCI scope: only the tokenization service's CDE zone requires the full PCI audit.

Network tokens (Visa and Mastercard's own token system): Visa and Mastercard now offer network-level tokenization (Visa Token Service). Instead of storing the real PAN, merchants store a network token specific to their merchant ID. Network tokens have better authorization rates (networks prefer them), automatic updates when physical cards expire, and are useless if stolen (they work only with the specific merchant). Stripe migrated all stored cards to network tokens in 2022.

Step 11: Reconciliation — Verifying the Ledger Is Correct

Even with idempotency keys, saga patterns, and careful coding, bugs happen. A deployment introduces a race condition that double-writes ledger entries for 0.001% of transactions for 2 hours before being caught. How do you find it?

Daily reconciliation:

Every night at 2am UTC:

1. INTERNAL RECONCILIATION (ledger balance check)
   For each currency:
     sum_of_all_credits = SELECT SUM(amount) FROM ledger_entries WHERE amount > 0;
     sum_of_all_debits  = SELECT SUM(amount) FROM ledger_entries WHERE amount < 0;
     
     assert sum_of_all_credits + sum_of_all_debits == 0
     -- If this fails: the ledger is not balanced. Immediate alert. Human review.

2. EXTERNAL RECONCILIATION (vs. card network statement)
   Visa sends a settlement file every morning:
     { "merchant": "merchant_abc", "date": "2024-01-15", "net_settled": 49823.10 }
   
   Platform's calculation from ledger:
     SELECT SUM(amount) FROM ledger_entries 
     WHERE account_id='merchant_abc' 
       AND entry_type='PAYMENT' 
       AND DATE(created_at)='2024-01-15';
   
   assert platform_ledger_sum == visa_settlement_amount ± $0.01
   -- Small rounding differences are acceptable; large differences require investigation.

3. BANK ACCOUNT RECONCILIATION (vs. actual bank statement)
   Platform's bank provides a daily statement.
   Compare: sum of all payouts submitted for settlement vs. bank debits.
   compare: sum of all card captures settled vs. bank credits.
   Any mismatch: alert and investigate.

Discrepancy categories:

• Timing differences: A payment captured on 11:59pm is in platform's ledger for Day 1 but Visa settles it on Day 2. Not an error — just a timing difference. Tracked by matching on the card network's transaction ID, not by date alone.
• True discrepancies: Platform charged the card but ledger was not credited (bug). Or ledger shows a capture that Visa doesn't acknowledge (bug in the authorization layer). These are real money discrepancies requiring investigation and manual correction.

Stripe's reconciliation system processes billions of records nightly and resolves discrepancies automatically for known patterns (timing differences) while escalating true discrepancies to an operations team within 30 minutes of the reconciliation run completing.

Step 12: Fraud Detection

Every payment must be screened for fraud before authorization. A fraudulent payment — where the card is being used by someone other than the cardholder — results in a chargeback: the cardholder disputes the charge with their bank, the bank reverses the funds, and the merchant loses both the goods and the money.

Fraud detection must run synchronously, within the payment's latency budget. Adding 500ms for fraud scoring is acceptable (total latency: 3.5 seconds). Adding 5 seconds is not (user abandons checkout).

Signals used for fraud scoring:

Card signals:
  - BIN (first 6 digits) — which bank, what country, credit vs debit
  - Card velocity: how many transactions on this card in the last hour? (>5 is suspicious)
  - Card freshness: just tokenized for the first time? (stolen cards are often used immediately)

User/merchant signals:
  - IP geolocation vs. card billing country match
  - Device fingerprint (browser, OS, screen resolution, fonts) — does it match prior purchases?
  - Email address age (brand new Gmail account + high-value purchase = suspicious)
  - Purchase amount vs. merchant category + customer history

Network signals:
  - Is this IP in a known proxy/VPN? (fraudsters use VPNs to hide location)
  - Has this IP been associated with prior chargebacks?
  - Is the shipping address different from billing address? (common in fraud)

Behavioral signals:
  - Time from page load to checkout (<3 seconds: bot; >10 minutes: human browsing)
  - Mouse movement patterns (bots move in straight lines)
  - Number of card attempts in this checkout session (trying multiple cards = carding attack)

The fraud score is a probability: P(fraud | signals). A rule-based system combines scores with thresholds: if score > 0.8, block. If score > 0.5, trigger 3DS (3D Secure — send the cardholder an OTP to their phone). If score < 0.5, allow.

3D Secure (3DS2): An authentication protocol where the card network (Visa Secure, Mastercard Identity Check) sends an OTP or biometric challenge to the cardholder's banking app. Completing 3DS shifts liability for fraud from the merchant to the card network — if the transaction is later disputed as fraudulent, the network (not the merchant) absorbs the chargeback. 3DS adds 15–30 seconds to checkout (user must approve in their banking app) and increases friction, but protects high-risk transactions.

Step 13: The Complete Architecture

Storage and the nightly reconciliation job behind those services:

Step 14: Failure Modes — The Correctness Checklist

For each failure scenario, trace the outcome:

Crash between ledger write and card network call: Payment record is PENDING in the database. Card was never charged. A recovery job finds PENDING payments older than 5 minutes and marks them FAILED (they were never sent to the card network). Merchant receives a failure webhook. No money moved. Safe.

Crash after card network authorization but before recording it: Card is authorized (cardholder's credit is held). Our database shows status=PENDING. Recovery job queries Visa using the payment_id: "Was this authorized?" Visa returns the auth code. Recovery job records the authorization and captures. Payment completes. Or: if Visa says "not authorized," mark FAILED and the auth was never completed. Safe.

Card network times out during capture (money in transit): Authorization completed. Capture request was sent. No response. Recovery job queries Visa: "Was this captured?" If yes: record capture, write ledger entries. If no: retry capture (idempotency key on the capture request prevents double capture). Safe.

Crash during saga step 3 (after card capture, before provider credit): $100 has been captured from the cardholder. Platform has the money. Provider has not been credited. The saga's durable state shows: steps 1 and 2 completed, step 3 not started. Recovery: execute step 3 (credit provider $90). The capture (step 2) is already done and should not be compensated — the customer's payment succeeded. Only incomplete forward steps need retry.

Double reconciliation run (idempotency in batch jobs): Nightly reconciliation runs twice due to a scheduler bug. If the reconciliation job is not idempotent — if running it twice creates duplicate discrepancy tickets or double-compensating corrections — the second run corrupts state. Fix: reconciliation jobs must check their own run history: "Has the reconciliation for 2024-01-15 already been marked complete? If yes, skip." Same idempotency-key concept, applied to batch jobs.

Wrong amount in ledger (software bug): A code bug causes fee_amount to be subtracted instead of added in the ledger entry for 2 hours. The ledger shows platform_revenue losing money instead of gaining it. The internal reconciliation catches this at 2am: sum_of_all_entries ≠ 0. Alert fires. Engineering is paged. The correction: add compensating ledger entries that reverse the incorrect entries and add correct ones. The original wrong entries are never deleted — they remain in the ledger with a note referencing the correction entries. The audit trail is complete.

Observability — What to Instrument

Payment systems require both technical observability (Chapter 18's metrics/traces/logs) and financial observability (reconciliation, balance dashboards).

Technical metrics:

payment_attempts_total{status="succeeded"|"declined"|"failed"}  (counter)
payment_latency_seconds{step="fraud"|"card_network"|"ledger"}  (histogram)
idempotency_key_hits_total{result="new"|"duplicate"|"in_flight"}  (counter)
card_network_errors_total{network="visa"|"mastercard", error_type="timeout"|"5xx"}  (counter)
saga_stuck_count  (gauge — alert if > 0)
reconciliation_discrepancy_amount_usd  (gauge — alert if abs > $1)

The single most important alert: reconciliation_discrepancy_amount_usd with threshold $0.01. Any discrepancy between the ledger and external settlement files must page someone immediately. This is the financial equivalent of Chapter 18's "dead man's switch" — the reconciliation that doesn't balance is the signal that something is fundamentally wrong.

Trace every payment end-to-end: Every payment has a payment_id that serves as the trace ID (Chapter 18). All logs, all spans, all external calls carry the payment_id. When a merchant reports "payment ch_xxx failed mysteriously," you pull the trace and see every step, its latency, its outcome, and the exact error from the card network. Payment investigation without complete traces is a multi-hour forensic exercise. With traces: 2 minutes.

The payment dashboard (not for engineers — for operations):

Real-time:
  - Authorization rate (% of attempts authorized, last 5 min)
  - Decline rate breakdown (insufficient funds, card invalid, fraud blocked)
  - P99 latency per step (fraud scoring, card network, ledger)
  - Stuck sagas count (must be 0)

Daily:
  - Gross payment volume ($)
  - Net revenue (fees earned)
  - Refund rate (high refund rate = merchant problem or fraud)
  - Chargeback rate (>1% of transactions = Visa/MC may suspend merchant)
  - Reconciliation status (balanced / discrepancy / pending)

Vocabulary Map

Practice Problems

Problem 1 — Failure Analysis

A merchant's server sends a POST /charges request. Your payment service receives it, acquires the idempotency key lock, calls the card network, receives AUTHORIZED (auth_code: V3F8K2), but crashes before writing the authorization to the payments table.

(a) What is the state of the world: what does the card network know, what does your database show, what does the merchant's server know?

(b) The merchant's server times out and retries with the same idempotency key. Trace the full flow: does the idempotency check catch the retry? Why not (hint: the idempotency key row was inserted but the response column is NULL — what does this mean)?

(c) Design the recovery mechanism: a background recovery job scans for payments in PENDING status with created_at > 5 minutes ago. It finds this payment. What query does it send to Visa? What does Visa return? What does the recovery job do with each possible response (authorized, not found)?

(d) The recovery job runs while the original payment service instance (which crashed and restarted) is also trying to recover the same payment simultaneously. Both instances query Visa, both receive AUTHORIZED. Both try to write the authorization to the database. Design the database-level concurrency control that prevents a double-authorization write. (Hint: what constraint on the payments table, combined with which SQL construct, prevents two concurrent writes from both succeeding?)

Problem 2 — Design from Scratch

Design the webhook delivery system: when a payment status changes (to SUCCEEDED, DECLINED, REFUNDED), the merchant receives an HTTP callback to their registered endpoint.

Requirements: at-least-once delivery (missed webhooks mean merchants miss order notifications), delivery retry with exponential backoff (Chapter 17's retry pattern), webhook signature verification (the merchant can verify the webhook came from your platform, not an attacker), delivery ordering per merchant (later events should not arrive before earlier events for the same payment), dead letter queue for permanently failing endpoints.

Specify: (a) how webhook events are produced (Kafka topic, partition key choice — and why payment_id not merchant_id and not random), (b) the retry schedule (how many retries, what backoff, when to give up), (c) webhook signature: what is signed, what algorithm, what is the HMAC key (per-merchant or global?), (d) how you guarantee ordering — if SUCCEEDED fires at T=0 and REFUNDED fires at T=5 minutes, the merchant must receive SUCCEEDED first. How does Kafka partition key choice enforce this? (e) what happens when a merchant's endpoint is down for 24 hours — how many retries accumulate, how is the backlog managed, and what does the merchant see when their endpoint recovers?

Problem 3 — Synthesis

Connect the payment system to Chapter 12's Raft and Chapter 6's consistency models.

The payment service uses PostgreSQL with synchronous replication (2 sync replicas). Every ledger write must be acknowledged by all 3 nodes before returning to the caller (Chapter 4/5's synchronous replication with synchronous_commit=on).

(a) A network partition separates the primary from both replicas. The primary can no longer write (cannot achieve quorum). Payment requests fail. Chapter 6's CAP theorem: which property is the system prioritizing (consistency or availability)? Is this the right choice for a payment system? What is the cost of choosing consistency here?

(b) An engineer suggests: "during the partition, allow the primary to continue writing without replica confirmation. We'll reconcile after the partition heals." Trace through what happens: primary writes 500 payments during the partition. Replicas have a different view. Partition heals. Both primary and replicas have conflicting ledger entries. How does PostgreSQL handle this (hint: Chapter 12's leader/follower concept — the primary is always authoritative, replicas discard their conflicting writes). Is any payment data lost? Is the ledger still balanced after reconciliation?

(c) If the system chose availability (the suggestion in (b)), what specific correctness invariants might be violated? Construct a concrete scenario involving a payout that is requested during the partition, resulting in a double-payout after reconciliation. Trace the exact ledger entries and show why the double-payout is a correctness violation.

Problem 4 — Go Deeper

Analyse the idempotency key system under adversarial conditions.

(a) An attacker discovers that your idempotency key system caches responses by key. They make a payment of $1 (a small test charge), record the idempotency key. They then construct a request for $10,000 using the same idempotency key. What does your system return? What validation prevents this from being a problem? (Hint: the request hash check — what does it compare and what does it return on mismatch?)

(b) A distributed idempotency system: your payment service runs on 10 instances. Two instances receive the same idempotency key simultaneously (a client sent two concurrent requests — network issue). Both instances query the idempotency_keys table. Both see "no existing key" (before either INSERT completes). Both proceed to charge the card. Double charge.

Solve this with PostgreSQL's INSERT ... ON CONFLICT DO NOTHING RETURNING *: if the INSERT returns a row, this instance "won" the race and should proceed. If it returns no rows (conflict — another instance already inserted), this instance should read the existing row and follow the cached-response path. Walk through the exact SQL for both the INSERT and the subsequent read. Is this guaranteed to be race-free in PostgreSQL? (Hint: what isolation level does PostgreSQL use by default, and does INSERT ... ON CONFLICT provide the atomic guarantee you need?)

(c) Idempotency key TTL: idempotency keys are stored forever (no deletion). At 11.6 requests/second and one key per request, how many keys accumulate per year? At 100 bytes per row, what is the storage cost after 10 years? Design a TTL policy: after what time is it safe to expire an idempotency key, and what is the risk of expiring it too early? (A merchant's retry logic might retry after 24 hours if their server was down — should 24-hour-old keys still be honored?)

Problem 5 — Connect to Previous Chapters

The payment system orchestrates across four chapters' worth of mechanisms. For each scenario, identify the exact mechanism from the cited chapter that applies and explain the connection.

(a) Chapter 2 (WAL) + Chapter 13 (Sagas): A payment saga is in step 3 (crediting the provider's balance). The saga orchestrator writes its step-3 state to PostgreSQL and is about to call the internal transfer API. The orchestrator crashes. When it restarts, it reads its state from PostgreSQL: step 3 was written as "in progress." How does this resemble WAL's role in crash recovery? What is the equivalent of WAL "redo" in the saga context?

(b) Chapter 14 (message queues) + idempotency: The webhook delivery system uses Kafka (Chapter 15) with at-least-once delivery. A webhook for "payment SUCCEEDED" is delivered to the merchant twice (Kafka redelivery after consumer restart — Chapter 15's at-least-once semantics). The merchant's webhook handler must be idempotent — processing the same event twice must produce the same result as processing it once. Design the merchant-side idempotency key: what field in the webhook payload uniquely identifies an event, and how should the merchant's handler use it to detect and skip duplicates?

(c) Chapter 17 (rate limiting) + fraud detection: A fraudster attempts a "carding attack": they have a list of 10,000 stolen card numbers and want to test which are still valid by making small $1 charges. At 10 attempts/second, they'll cycle through all cards in 17 minutes. Design the multi-layer rate limiting that stops this attack. What dimensions do you rate-limit (per IP? per card BIN? per merchant? per device fingerprint?)? At what threshold does each layer trigger? What is the response when the limit is triggered (reject with 429, or silently approve but flag for fraud review)? Explain why silently flagging (without a 429) is sometimes better than a hard reject for fraud defense.

(d) Chapter 18 (observability) + reconciliation: You are on call. At 3:17am, the reconciliation job fires an alert: "Ledger imbalance detected: $47,231.00 discrepancy." You have 30 minutes to determine whether this is a real money error or a data pipeline issue before the CFO's morning report.

Walk through the exact observability steps: which metrics do you look at first, which Kafka consumer lag tells you if the reconciliation pipeline is stale, which SQL query on the ledger tells you which account is out of balance, which payment_events records help you find the root cause. Reference Chapter 18's "observability workflow" (alert → dashboard → trace → logs) applied to this financial incident.

What's Coming Next

Chapter 24 is the final chapter: the search engine. It synthesises the storage layer (inverted indexes — a completely different data structure than B-Trees or LSM trees), the distributed systems layer (sharding an inverted index, coordinating distributed search queries), and the ranking layer (how relevance scores are computed, stored, and sorted across billions of documents). Search is everywhere — Google, Elasticsearch, Postgres's tsvector, the search bar in your SaaS product — and building it from first principles closes the loop on the entire curriculum. After Chapter 24, you will have designed eleven complete distributed systems from scratch, from first principles. That is the foundation.

Chapter 23 complete. The payment chapter is different from every other chapter in the curriculum: the bottleneck is not throughput, not latency, not scale — it is correctness. Every mechanism (idempotency keys, double-entry ledger, saga pattern, synchronous replication, reconciliation) exists to enforce one invariant: money is conserved. Problem 4b (the concurrent idempotency key race) is the subtlest correctness problem in the curriculum — work through the exact SQL and the PostgreSQL isolation guarantee before moving on. Problem 5d (the 3am reconciliation incident) is the closest thing to a real on-call experience this curriculum provides — trace it step by step, as if it were real.