Chapter 8 — Observability and Recovery

See the failure fast, undo it safely, change the schema without stopping traffic, and practise all of it before it happens

Prerequisites: SD 2 Ch 18 — Observability (metrics, traces, logs, SLOs, burn-rate alerts, runbooks), Debugging 10 — Production observability (p50 vs p99, RED and USE, alert to root cause), Debugging 15 — Incidents and postmortems (mitigation levers, why rollback is not always safe, postmortems), PostgreSQL From First Principles, Ch 18 (migration lock costs in outline), SD 1 — Load balancing (rolling, blue-green and canary deploys).

How to read this chapter

Observability, SLOs, rollbacks and postmortems are covered in depth in the chapters above, so Parts 1, 2 and 7 are summaries: what you must be able to say in the room, with links. The rest of the chapter is about two things the reader is thin on:

• Zero-downtime schema changes (Parts 3 to 6): expand/contract, the order of deploys, which Postgres DDL blocks traffic and for how long, and backfills that don't lock the table.
• Fault injection, chaos testing and game days (Part 8): breaking your own system on purpose, safely, so that the interviewer's "what if X dies?" has an answer you have measured.

Every Postgres number was measured on PostgreSQL 18.6 on a laptop, against a 5-million-row table (510 MB). The Node code was run on Node 22 with pg 8.22. Absolute times scale with table size and hardware. Which statements block what does not.

Part 1: Observability in the room

What you must be able to say (Debugging 10 and SD 2 Ch 18 have the detail):

• Watch p50 and p99 together. p50 rising means everyone is slower: look for a shared cause. p99 rising alone means some requests are slower: break it down by endpoint, tenant, host and version. You can't average percentiles across servers. And tails multiply: a page that waits on 10 services, each slow 1% of the time, is slow on 1 − 0.99¹⁰ ≈ 9.6% of loads.
• SLIs measure what users experience. SLOs set the target. The error budget is what's left to spend. For 99.9% over 30 days, that's 43.2 minutes, or 0.1% of requests.
• Alert on burn rate, not on thresholds. A 14.4× burn over the last hour (2% of the monthly budget gone in 60 minutes) pages someone now. A 1× to 2× burn over 6 hours opens a ticket. Cause metrics such as CPU and memory are dashboard clues, not pages.
• Saturation is the early warning: pool wait count, queue depth, event-loop delay. They rise minutes before errors do.
• Every alert maps to a first runbook step. If the on-call engineer's first move isn't written down, the alert is noise.

Part 2: Rollbacks and feature flags

• Most incidents start with a change. The fastest mitigation is usually to undo the change: roll back the deploy, or turn off the flag. Reading the diff comes later.
• A flag is faster than a rollback. A flag flip takes seconds and doesn't need a build. It only helps if the broken code is behind that flag. The kill-switch design, including what the flag defaults to when the flag service itself is down, is in Chapter 6, Part 4.
• Canary first. Send 1–5% of traffic to the new version, compare its error rate and p99 with the old version's, and promote it only if they match (SD 1 — Load balancing). A bug that hits 1% of requests on a 5% canary fails 0.05% of total traffic.
• A rollback is only safe if the old code works against the current schema and data. If the release dropped a column, renamed one, or wrote rows in a format the old code can't read, rolling back the code makes things worse. That is the whole reason for Parts 3 to 6.

Part 3: Why schema changes break running systems

A schema change is hard to do safely because three things are true at once:

1. Two versions of the code run at the same time. A rolling deploy of 6 pods runs v1 and v2 side by side for minutes, and a rollback does the same in reverse. The schema at any moment has to work for both.
2. The migration and the code ship separately. The migration runs once, at some moment, and the pods change over time. Whichever runs first, the other one has to cope.
3. DDL takes locks, and in Postgres a lock that waits can stop traffic even when the DDL itself would take 0.3 ms (Part 5).

Here is what happens when you ignore rule 1. The release renames a column in a migration that runs before the pods are replaced:

Expand/contract

The fix is to split one breaking change into several compatible ones. Renaming users.name to users.full_name:

Notice that every step except the last can be undone by a normal rollback. Notice also that a "rename" became four deploys and two migrations. That's the price. Smaller teams sometimes merge steps 2 and 5 or 5 and 6. What you can never merge is the expand step with the contract step.

The same pattern covers changing a column's type (add the new column, dual write, backfill, switch, drop), splitting a table, and moving data to a new store.

Dual writes: in one database, and across two

Inside one Postgres database, the dual write happens in the same transaction as the original write, so the two columns can't drift apart. A trigger can do the copy instead of the app (BEFORE INSERT OR UPDATE … SET NEW.full_name = NEW.name). This works even for writers you forgot about, such as admin scripts and other services, at the cost of hidden logic that must be removed in the contract step.

Across two systems (Postgres to a new search index, or an old service's database to a new one), a dual write from the app is not atomic. The first write succeeds, the process crashes, the second never happens, and the stores disagree with nothing recording the gap. Use the outbox pattern or change data capture instead: write once, and let a reliable consumer copy the change (Q49 — Outbox). Then run a reconciliation job that compares the stores and reports differences, before you switch reads.

Part 4: The order of deploys

Four rules turn expand/contract into a release plan:

1. Schema before code that needs it. The expand migration ships and completes before any pod that writes the new column starts.
2. Code stops using something before the schema removes it. The contract migration ships in a later release than the code change that stopped reading and writing the old column, after that release has soaked.
3. Every release must work with the schema of the release before it and the one after it. That's what makes both the rolling deploy and the rollback safe.
4. Migrations run as their own step, once, not on every pod's boot. Six pods booting at once means six runners racing. Most migration tools take a lock so that only one proceeds, but a separate pipeline step also gives you one place to watch, time and stop the migration.

One more trap at the contract step: code you don't think of as reading the column. ORMs that SELECT * into a model with a fixed field list, reports and analytics jobs, other services sharing the database, and prepared statements that can fail with cached plan must not change result type after a column is dropped. Search query logs (pg_stat_statements) for the column name before you drop it.

Part 5: Lock-safe Postgres DDL, measured

Almost every ALTER TABLE takes the strongest lock

The surprising result first: adding a nullable column takes an ACCESS EXCLUSIVE lock, the same mode as a full table rewrite. It blocks every read and every write on the table while it is held. The safe forms are safe because they hold that lock for well under a millisecond, not because they take a weaker lock. (The PostgreSQL chapter's table lists ADD COLUMN under AccessShareLock. The run below shows it is AccessExclusiveLock, held briefly.)

Method: each DDL ran inside BEGIN … ROLLBACK on a 5-million-row, 510 MB table and held its lock for 2 s. pg_locks showed the mode. A second session tried a primary-key SELECT and an INSERT, each with lock_timeout = '300ms'. The time is how long the DDL itself took, which is how long an autocommitted DDL would hold the lock.

The full scans took about 100 ms here because the 510 MB table was in memory. On a 500 GB table read from disk, the same SET NOT NULL holds ACCESS EXCLUSIVE for minutes.

Recipes

Add NOT NULL to an existing column without a long lock (Postgres 12 and later):

SET lock_timeout = '2s';
ALTER TABLE t ADD CONSTRAINT a_not_null CHECK (a IS NOT NULL) NOT VALID;  -- 0.6 ms
ALTER TABLE t VALIDATE CONSTRAINT a_not_null;     -- 103 ms, reads and writes keep running
ALTER TABLE t ALTER COLUMN a SET NOT NULL;        -- 0.4 ms: no scan, the check proves it
ALTER TABLE t DROP CONSTRAINT a_not_null;         -- optional, now redundant

With the valid check in place, SET NOT NULL dropped from 116 ms to 0.4 ms, and at client_min_messages = debug1 Postgres says why: "existing constraints on column "t.a" are sufficient to prove that it does not contain nulls". Postgres 18 also accepts ADD CONSTRAINT … NOT NULL a NOT VALID directly. That was tested too, but only works on 18.

Add an index: CREATE INDEX CONCURRENTLY. It builds in several passes and waits for transactions that started before it, so a long-running transaction delays it, though it never blocks other sessions. If it fails or is cancelled, it leaves an invalid index behind. Cancelled with statement_timeout, the test left t_b_idx with indisvalid = false. The invalid index still costs time on every write and is not used by queries. Drop it with DROP INDEX CONCURRENTLY and try again.

Add a foreign key: ADD CONSTRAINT … NOT VALID, then VALIDATE CONSTRAINT in a separate statement.

Change a column's type: don't use ALTER COLUMN TYPE on a big table. Add a new column, dual write, backfill, switch, drop the old one (Part 3). A few changes don't rewrite the table, such as increasing a varchar(n) limit or changing varchar to text. Test on a copy if you're not sure.

The lock queue: why an instant DDL can stop your API

The table above says ADD COLUMN takes 0.3 ms. Here it is taking most of 5 seconds of traffic with it:

The holder doesn't have to be a report. Anything that has touched the table inside an open transaction counts: a slow analytics query, a connection left idle in transaction, a long backfill, an autovacuum running to prevent wraparound (that one doesn't step aside for you). So migrations always run with a lock_timeout and a retry loop, and teams also set idle_in_transaction_session_timeout so a forgotten transaction can't hold a lock for hours:

import { setTimeout as sleep } from 'node:timers/promises';

// Run one DDL statement without letting it sit in the lock queue.
// 55P03 = lock_not_available, the error lock_timeout raises.
export async function runDdl(pool, sql, { lockTimeout = '2s', attempts = 10 } = {}) {
  for (let attempt = 1; ; attempt++) {
    const client = await pool.connect();
    try {
      await client.query(`SET lock_timeout = '${lockTimeout}'`);
      await client.query(sql);
      return attempt;
    } catch (err) {
      if (err.code !== '55P03' || attempt === attempts) throw err;
      const wait = Math.min(30_000, 250 * 2 ** attempt) * (0.5 + Math.random() / 2);
      console.warn(`attempt ${attempt}: no lock within ${lockTimeout}, retry in ${Math.round(wait)} ms`);
      await sleep(wait);
    } finally {
      await client.query('RESET lock_timeout').catch(() => {});
      client.release();
    }
  }
}

Tested with a transaction holding the table for 5 s and lockTimeout: '1s': attempts 1 to 3 timed out, the holder committed at 5.0 s, and attempt 4 succeeded at 5.4 s. Each attempt could hold up other queries on the table for at most the 1 s lock_timeout.

Part 6: Backfills in batches

A backfill fills the new column for existing rows. The tempting version is one statement:

UPDATE t SET region = 'eu';   -- 5,000,000 rows

Measured on the 5-million-row table, it ran for 22 to 33 s. While it ran:

• Every row it had touched stayed locked until it committed. A single-row UPDATE from another session, with lock_timeout = '300ms', was blocked on rows the big update had reached. In an app, that's every write to those rows waiting up to 30 s.
• It wrote 1.7 GB of WAL in one transaction, which replicas must receive and replay, so replica lag grows by the whole amount.
• The table grew from 510 MB to 1,020 MB, because MVCC keeps the old version of every row until vacuum can remove it, and it can't while the transaction is open.
• If it fails at row 4,900,000, all of it rolls back.

The batched version commits every 50,000 rows by primary key range. Each batch took about 150 ms. The concurrent single-row updates all succeeded. The whole backfill took 41.7 s, almost twice as long. That's the trade-off: batches are slower in total, but each one holds its locks for a fraction of a second, they can be paused and resumed, and replicas can keep up between them.

import { setTimeout as sleep } from 'node:timers/promises';

// Copy name -> full_name in small committed batches, walking the primary key.
// Safe to stop and rerun: rows already copied are skipped.
export async function backfill(pool, { batch = 5_000, pauseMs = 20, maxLagBytes = 64 * 2 ** 20 } = {}) {
  let lastId = 0;
  let updated = 0;
  for (;;) {
    const { rows } = await pool.query(
      `WITH b AS (SELECT id FROM users WHERE id > $1 ORDER BY id LIMIT $2),
            u AS (UPDATE users SET full_name = users.name
                  FROM b WHERE users.id = b.id AND users.full_name IS NULL
                  RETURNING 1)
       SELECT (SELECT max(id) FROM b) AS last_id, (SELECT count(*) FROM u)::int AS n`,
      [lastId, batch],
    );
    if (rows[0].last_id === null) return updated;
    lastId = rows[0].last_id;
    updated += rows[0].n;

    // Throttle: wait while any replica is far behind.
    for (;;) {
      const lag = await pool.query(
        `SELECT coalesce(max(pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn)), 0)::bigint AS bytes
         FROM pg_stat_replication`,
      );
      if (Number(lag.rows[0].bytes) < maxLagBytes) break;
      await sleep(1_000);
    }
    await sleep(pauseMs);
  }
}

What makes it safe, line by line:

• Keyset batches (id > $1 ORDER BY id LIMIT $2) use the primary key index, so every batch costs the same. OFFSET would get slower with every batch.
• full_name IS NULL makes it idempotent. Rerun it after a crash and it skips the rows already done. Rows that new code has dual-written are skipped too.
• Each batch is its own transaction, so locks are held for one batch only, and vacuum can clean up between batches.
• The lag check pauses when any replica is more than 64 MB behind. Without it, the backfill can push read replicas seconds behind and users start seeing stale data (Chapter 5).

Tested on 1 million rows: 24.0 s with 5,000-row batches and a 20 ms pause, a rerun updated 0 rows, and no full_name was left empty.

Part 7: Postmortems that change the system

Debugging 15 covers running the incident and writing the postmortem. In a design round you only need the principle: a postmortem is measured by what it changes in the system, not by what it says about people. Action items that work change a mechanism: a lock_timeout added to the migration runner, an alert on archive failures, a row limit on the cleanup job, a restore drill on the calendar. Action items such as "be more careful with migrations" change nothing. Each item has an owner and a due date, and the team tracks them to done. The best items can be tested, which leads to Part 8.

Part 8: Fault injection, chaos testing and game days

Every chapter in this section ends with "when the interviewer breaks it". In production, nobody tells you in advance. Fault injection is breaking it yourself, on purpose, at a time you choose, while people are watching. It turns "we have failover" into "failover took 41 s and lost 0 writes last Tuesday".

Where you can inject faults

The in-process version is a few lines of code and is the easiest place to start:

import { setTimeout as sleep } from 'node:timers/promises';

// Faults are read on every call, so a flag or admin endpoint can change them live.
// Shape: { recommendations: { latencyMs: 2000, errorRate: 0.5, percent: 10 } }
let faults = {};
export const setFaults = (next) => { faults = next; };

export function withFaults(name, fn) {
  return async function (input, { signal } = {}) {
    const f = faults[name];
    // percent limits the blast radius: only that share of calls is touched
    if (f && Math.random() * 100 < (f.percent ?? 100)) {
      if (f.latencyMs) await sleep(f.latencyMs, undefined, { signal });
      if (Math.random() < (f.errorRate ?? 0)) {
        throw Object.assign(new Error(`injected fault in ${name}`), { code: 'EINJECTED' });
      }
    }
    return fn(input, { signal });
  };
}

With { latencyMs: 2000, percent: 10 } on a recommendations call that the home feed wraps in a 300 ms timeout and an empty-list fallback, 1,000 concurrent feed requests all finished in 317 ms, and 105 of them rendered without recommendations. That's the experiment's result: the 300 ms timeout and fallback held. With { errorRate: 0.5 }, 481 of 1,000 fell back. Because the injected delay takes the caller's signal, the timeout cancels the sleep instead of leaving it running.

Running a chaos experiment

1. State the steady state as a metric. "Checkout success rate ≥ 99.5% and p99 < 800 ms, measured each minute."
2. Write the hypothesis with numbers. "If the Redis primary is killed, Sentinel promotes a replica within 30 s, and the steady state holds, because product pages fall back to Postgres."
3. Limit the blast radius. Staging first. Then production during working hours, on one cell, one zone or 1% of traffic, never everything at once.
4. Set abort conditions before you start. "If checkout success drops below 98% for 60 s, stop the experiment," and know exactly how to stop it: a flag, a command, a person with their hand on it.
5. Run it, watch it, and write down what really happened, including what you didn't predict.
6. Turn every surprise into an action item, and run the experiment again once it's fixed.

Don't run chaos experiments on things you already know are broken. If there's no timeout on the payment call, the experiment teaches you nothing you don't know, and it hurts users. Fix the known gaps, then test that the fixes work.

Try it in the sim

The load-balancer sim works like a small chaos experiment. In the Server dies tab, server C crashes at 2.2 s while health checks probe every 0.5 s and eject a server after 3 failed probes. Predict what clients see, then run it and compare with the "Failed · no checks" count, the same run without health checks. Then switch to the Slow server tab: B becomes 5× slower but stays up and keeps passing health checks. That's the fault your health checks don't catch, which is why latency injection belongs in every experiment plan. In free play you can click any server to cycle it between healthy, slow and dead.

A game day, step by step

A game day tests the people and the process as well as the system. The roles: a facilitator who injects the failure and knows the scenario, responders who use only the real alerts, dashboards and runbooks, and an observer who writes down times. Here is a failover game day in staging, under a load test:

Each finding becomes an action item: a faster burn-rate alert for writes, an updated runbook, and pool settings that drop connections on errors and re-resolve DNS (Chapter 5 covers the failover side). Then the team runs the same game day again. Repeating it is how you find out whether the fixes worked.

What to put on the list

Map each chapter of this section to at least one experiment:

When the Interviewer Breaks It

Try each one out loud before opening the answer.

Break 1: "Rename users.name to full_name. 200 million rows, no downtime."

Break 2: "Your migration only added a nullable column. Why did the API time out for a minute?"

Break 3: "The deploy is bad. Roll back. Oh, and the migration dropped a column."

Break 4: "How do you know your failover actually works?"

Break 5: "What do you alert on, and what does on-call do when it fires at 3 a.m.?"

The Causal Chains

Checklist: add this to any design

1. SLOs per user journey, with burn-rate alerts that page and a runbook whose first step is "what changed?".
2. Deploy markers, canaries and feature flags, so a bad release is seen in minutes and undone in seconds.
3. Expand/contract for every schema change: no RENAME, DROP or type change in the same release as the code that depends on it.
4. One migration step in the pipeline, run once, with lock_timeout (2–5 s) and retries, and a restore point created before it.
5. Lock-safe forms: CREATE INDEX CONCURRENTLY, constraints NOT VALID then VALIDATE, SET NOT NULL through a validated check, new column + backfill instead of ALTER COLUMN TYPE.
6. Batched, idempotent, throttled backfills, by primary key, with a verification query before reads switch.
7. Timeouts that stop forgotten transactions: idle_in_transaction_session_timeout, and statement_timeout for reporting roles.
8. A fault-injection hook in each client (latency, errors, percentage), off by default, so experiments don't need a deploy.
9. A game day calendar: failover, cache loss, slow downstream, backup restore, each with a numeric hypothesis and abort conditions, rerun after fixes.

What's next

That's the last topic chapter. The drills put them together under pressure: a design is drawn, then broken step by step. Start with Drill 9 — The Cache Dies. Drill 14 — The Bad Deploy uses this chapter and Chapter 7 most: a bad migration, a rollback that won't work, and a restore that has never been tested.