Chapter 10 — Reviewing a Codebase and Defending Your Designs

A method for unfamiliar code, a way to rank and say what you find, and the questions they will ask about what you built

Prerequisites: Chapters 03–09, especially 05 — Race Conditions, 07 — Controlling Concurrency and 08 — The Node Process Under Failure. Related: Debugging 13 — Slow, Flaky and Falling Over.

How to Read This Chapter

The earlier chapters each taught one family of bugs. This one is about the interview session itself: you are handed a repository you have never seen, told "review this", and given 30 to 45 minutes. Later, or in another round, you are asked about systems you built, and the interviewer keeps asking "why" until they find the edge of your understanding.

Both halves test the same skill: explaining a mechanism out loud. "This is racy" scores little. "Two requests both read 100 here, both pass the check on line 54, and both write 20 on line 64, so one charge is lost" scores well, because it proves you can see the interleaving.

Part 1 gives a fixed order of things to look at. Part 2 shows how to rank and phrase what you find. Part 3 is the design vocabulary you need to explain why code is shaped wrong. Part 4 is a full worked review: try it before opening the answer. Part 5 is the "defend your design" drill. Part 6 is a table of phrases.

Part 1: The First 15 Minutes in an Unfamiliar Repo

Do not read the code top to bottom. Reading one path in order shows you what the code does when nothing else is happening, and concurrency bugs live in what happens when something else is happening. Instead, look for the places where bugs live, in this order. Say each step out loud as you do it; the interviewer is scoring the method as well as the findings.

1. Entry points and data flow. Find where requests, messages and timers enter. For each, trace where the data goes: which tables, which caches, which outside services. Draw it in one line ("HTTP → price cache → wallet → orders → provider → orders"). You now know what can run at the same time as what.

2. Shared mutable state. Anything at module level that changes is shared by every request in the process, and not shared between processes. Ask two questions of each: who else writes this between my read and my write, and what happens with two instances behind a load balancer?

3. Every external call. For each fetch, query or SDK call, ask four questions:

• Timeout. What happens if it never answers? fetch has no default timeout.
• Retry. Is it retried? With backoff and jitter? Which errors?
• Idempotency. If it is retried, or the client retries, does the effect happen twice? Is there a key?
• Cancellation. If the caller gives up, does the work stop, or does it keep running and costing money?

4. Every await between a read and a write. This is the single most productive search. Each await is a point where other tasks run (Chapter 01). A read, an await, then a write based on what was read, is the same race as SELECT then UPDATE in two Postgres transactions without a lock (Chapter 05).

5. Error paths and process lifecycle. Read every catch. Does it swallow, log without context, or rethrow without cause? Look for floating promises. Check what happens on SIGTERM during a deploy: are in-flight requests finished, are timers holding work that disappears? (Chapter 08.)

6. The tests. What is tested tells you what the author worried about. What is missing tells you where to look. A test suite with no concurrent test for a money path means nobody has checked the race.

Part 2: Triage and Presenting Findings

Rank by blast radius

You will find more problems than you have time to discuss. Lead with the ones that cost the most when they happen.

Say the ranking out loud: "I found about ten things. Two can lose money, so I'll start there."

Phrase each finding in four parts

1. Symptom: what a user or operator would see.
2. Mechanism: the exact interleaving or failure, with line numbers.
3. Fix: the smallest change that removes the mechanism.
4. Trade-off: what the fix costs or does not cover.

"Under concurrent orders for the same user, the wallet can go negative and charges can be lost. Two requests both read balance 100 on line 53, both pass the check on line 54, and both write balance - cost on line 64, so the second write overwrites the first. The fix is one conditional update, SET balance = balance - $1 WHERE user_id = $2 AND balance >= $1, and checking rowCount. That moves the check into the row lock. It does not stop a double-clicked submit from creating two orders; that needs an idempotency key."

That answer took 25 seconds and covered everything an interviewer looks for.

When not to rewrite

The urge in a review is to say "I'd rewrite this whole module." Resist it, for three reasons: you do not know the constraints that shaped it, a rewrite delivers nothing until it is done, and it reads as not being able to work with existing code.

• Patch when the design is sound and the bug is local: a missing await, a check-then-act that becomes one SQL statement, a missing timeout.
• Name the redesign, then patch when the abstraction itself is the bug. A setTimeout-based job runner cannot be made durable by editing it; work held in memory dies with the process. Say: "The short-term fix is X. The real fix is moving this to the jobs table, which is a bigger change I'd plan separately."
• Leave it when it is ugly but correct and the change carries more risk than the problem. Say so explicitly; it shows judgement.

Part 3: Design Critique Vocabulary

Finding a bug is half the job. The other half is saying why the shape of the code made the bug likely, and what shape would make it unlikely. These are the terms to use, each with the JavaScript form it takes.

Who owns this state?

Every piece of mutable state should have one owner that decides what writes are allowed. When three functions all do wallet.balance -= x, nobody owns the balance, so nobody can enforce "never below zero". In a backend, the owner of money and job state should usually be the database, through constraints and conditional updates, because it is the one place every instance and every recovery path goes through.

Ask: "If I want to add a rule about this value, how many places do I have to change?" More than one means no owner.

Mutation vs immutability

JavaScript passes objects by reference. A function that mutates an argument changes the caller's object, and a module-level object that gets mutated changes it for every later request.

Drill 1 — The request defaults

const DEFAULTS = { model: 'standard', temperature: 0.7, stop: [] };

export function buildRequest(userOptions) {
  const opts = Object.assign(DEFAULTS, userOptions);
  if (opts.json) opts.stop.push('\n\n');
  return { ...opts, stream: true };
}

What's wrong?

export function buildRequest(userOptions) {
  const opts = { ...structuredClone(DEFAULTS), ...userOptions };
  if (opts.json) opts.stop = [...opts.stop, '\n\n'];
  return { ...opts, stream: true };
}

Module-level singletons and hidden coupling

A module is evaluated once per process, so const cache = new Map() at the top of a file is a process-wide singleton. That gives three hidden couplings:

• Between requests and tenants: one user's entry is visible to another's request unless the key includes the user.
• Between instances: two processes have two maps. An invalidation in one does not reach the other.
• Between tests: every test that imports the module shares the map, so test order changes results.

A module that does work at import time (setInterval, app.listen, opening a connection) couples every importer to that work. You cannot import it in a test without starting a timer that keeps the test process alive.

Drill 2 — The feature-flag cache

// flags.js
import { db } from './db.js';

let flags = null;
let loadedAt = 0;

export async function isEnabled(flag) {
  if (!flags || Date.now() - loadedAt > 60_000) {
    const rows = await db.query('SELECT name, enabled FROM flags WHERE tenant_id = $1', [currentTenant()]);
    flags = Object.fromEntries(rows.map((r) => [r.name, r.enabled]));
    loadedAt = Date.now();
  }
  return flags[flag] === true;
}

What's wrong?

const cache = new Map(); // tenantId -> { promise, expires }

export async function isEnabled(tenantId, flag) {
  let entry = cache.get(tenantId);
  if (!entry || entry.expires < Date.now()) {
    const promise = db.query('SELECT name, enabled FROM flags WHERE tenant_id = $1', [tenantId])
      .then((rows) => Object.fromEntries(rows.map((r) => [r.name, r.enabled])));
    entry = { promise, expires: Date.now() + 60_000 };
    cache.set(tenantId, entry);
    promise.catch(() => { if (cache.get(tenantId) === entry) cache.delete(tenantId); });
  }
  return (await entry.promise)[flag] === true;
}

Error-handling strategy

Three questions decide whether errors in a codebase are useful:

1. Error types. Can a caller tell a retryable failure from a permanent one without parsing the message? Use classes or a code field.
2. cause. When you wrap an error, keep the original: throw new OrderError('charge failed', { cause: err }). Node prints the whole chain. throw new Error(err.message) drops the stack, the type and every field on the original.
3. Where to catch. Catch where you can do something: retry, fall back, compensate, or turn it into a response. Catching in the middle only to log and rethrow creates duplicate log lines; catching to log and not rethrow hides failures.

export class ProviderError extends Error {
  constructor(message, { status, retryable, cause } = {}) {
    super(message, { cause });
    this.name = 'ProviderError';
    this.status = status;
    this.retryable = retryable;
  }
}

// at the call site
if (!res.ok) {
  throw new ProviderError(`provider ${res.status}`, {
    status: res.status,
    retryable: res.status === 429 || res.status >= 500,
  });
}

Never throw 'string': a string has no stack, and err instanceof Error checks fail.

EventEmitter vs promises vs queues

These look interchangeable in small code and are not.

The review smell: emitter.emit('order.paid', order) used to trigger the fulfilment of a paid order. If the process dies between the COMMIT and the listener finishing, the order is paid and never fulfilled, and nothing records that it was supposed to happen.

In-process vs durable work

setTimeout(() => sendInvoice(id), 0), void doLater(), an in-memory array used as a queue: all of these hold work in process memory. A deploy, a crash, or an out-of-memory kill drops it with no record.

The durable version writes the intent in the same transaction as the change that requires it:

await db.tx(async (t) => {
  await t.query("UPDATE orders SET status = 'paid' WHERE id = $1", [orderId]);
  await t.query(
    "INSERT INTO outbox (topic, payload) VALUES ('order.paid', $1)",
    [JSON.stringify({ orderId })],
  );
});
// A dispatcher reads `outbox` and enqueues or performs the side effect,
// marking each row done. If it crashes, it reads the same rows again.

The trade-off is at-least-once delivery: the dispatcher can send an item, crash before marking it done, and send it again. So the consumer must be idempotent. In-process work is at-most-once: it never duplicates, and it can vanish.

Leaky abstractions

An abstraction leaks when callers must know about what it hides in order to use it correctly. Common JavaScript forms:

• A cache.get(key) that is synchronous for hits and returns a promise for misses. Every caller has to handle both, and the ones that do not break only on cold starts.
• A retry(fn) helper that hides the retries, so callers do not know they must pass an idempotency key.
• A repository save(user) that silently does a read-modify-write, so two concurrent save calls lose an update, but nothing in its signature says so.
• A withTransaction helper that takes a callback but lets the callback use the global pool instead of the transaction client, so half the queries run outside the transaction.

Say it as: "This interface hides X, but callers still have to know about X to be correct, so it should either expose X or handle it."

Testability: inject the clock, fetch and the database

Code that reads Date.now(), calls the global fetch and imports a pool singleton can only be tested by monkey-patching globals or running real services. The fix is to pass dependencies in. That is all dependency injection means here.

export function createOrderService({ db, fetch = globalThis.fetch, now = Date.now, emailer }) {
  return {
    async placeOrder(userId, input) { /* uses db, fetch, now() */ },
    async refund(orderId) { /* ... */ },
  };
}

// main.js wires the real things; the test wires fakes:
const service = createOrderService({ db: fakeDb, fetch: fakeFetch, now: () => 1_700_000_000_000 });

The test can now control time, make the provider fail on the third call, and run two placeOrder calls concurrently against a fake database whose query awaits a controllable promise, which is how you write a deterministic test for a race.

Part 4: A Full Worked Review

The file below is a realistic order and credits service: Express 4, pg, a paid generation provider. It has about a dozen problems across every category above. Give yourself 20 minutes. Use the Part 1 order, write down each finding in the four-part form, rank them, and only then open the answer.

// orders.js — orders and credits for the generation API (Express 4, pg).
import express from 'express';
import { pool } from './db.js';
import { sendReceiptEmail } from './email.js';

export const app = express();
app.use(express.json());

const PROVIDER_URL = process.env.PROVIDER_URL;
const DEFAULT_OPTIONS = { model: 'standard', maxTokens: 1024, tags: [] };

const priceCache = new Map(); // userId -> Promise<price rows>
const jobStatus = {};         // orderId -> 'running' | 'done' | 'failed'

async function getPrice(userId, model) {
  if (!priceCache.has(userId)) {
    priceCache.set(
      userId,
      pool.query('SELECT model, credits FROM price_lists WHERE user_id = $1', [userId])
        .then((r) => r.rows),
    );
  }
  const prices = await priceCache.get(userId);
  return prices.find((p) => p.model === model).credits;
}

async function getBalance(userId) {
  const { rows } = await pool.query('SELECT balance FROM wallets WHERE user_id = $1', [userId]);
  return rows[0]?.balance ?? 0;
}

async function callProvider(prompt, options) {
  for (let attempt = 1; attempt <= 5; attempt++) {
    try {
      const res = await fetch(`${PROVIDER_URL}/v1/generate`, {
        method: 'POST',
        headers: { 'content-type': 'application/json' },
        body: JSON.stringify({ prompt, ...options }),
      });
      if (!res.ok) throw new Error('provider error ' + res.status);
      return await res.json();
    } catch (err) {
      console.log('provider failed, retrying', attempt);
    }
  }
}

app.post('/orders', async (req, res) => {
  const userId = req.user.id;
  const options = Object.assign(DEFAULT_OPTIONS, req.body.options);

  const cost = await getPrice(userId, options.model);
  const balance = await getBalance(userId);
  if (balance < cost) {
    return res.status(402).json({ error: 'insufficient credits' });
  }

  const { rows } = await pool.query(
    `INSERT INTO orders (user_id, prompt, cost, status)
     VALUES ($1, $2, $3, 'pending') RETURNING id`,
    [userId, req.body.prompt, cost],
  );
  const orderId = rows[0].id;
  await pool.query('UPDATE wallets SET balance = $1 WHERE user_id = $2', [balance - cost, userId]);

  jobStatus[orderId] = 'running';
  res.status(202).json({ orderId });

  setTimeout(() => runOrder(orderId, userId, req.body.prompt, options, cost), 0);
});

async function runOrder(orderId, userId, prompt, options, cost) {
  try {
    const result = await callProvider(prompt, options);
    await pool.query(
      "UPDATE orders SET status = 'done', output = $1 WHERE id = $2",
      [result.text, orderId],
    );
    jobStatus[orderId] = 'done';

    options.tags.forEach(async (tag) => {
      await pool.query('INSERT INTO order_tags (order_id, tag) VALUES ($1, $2)', [orderId, tag]);
    });
    sendReceiptEmail(userId, orderId);
  } catch (err) {
    jobStatus[orderId] = 'failed';
    await refund(orderId, userId, cost);
  }
}

async function refund(orderId, userId, amount) {
  const { rows } = await pool.query('SELECT status FROM orders WHERE id = $1', [orderId]);
  if (rows[0].status === 'refunded') return;

  await pool.query('UPDATE wallets SET balance = balance + $1 WHERE user_id = $2', [amount, userId]);
  await pool.query("UPDATE orders SET status = 'refunded' WHERE id = $1", [orderId]);
}

// Refund orders that got stuck (provider outage, crash, etc.).
setInterval(async () => {
  const { rows } = await pool.query(
    `SELECT id, user_id, cost FROM orders
     WHERE status = 'pending' AND created_at < now() - interval '10 minutes'`,
  );
  for (const order of rows) {
    await refund(order.id, order.user_id, order.cost);
  }
}, 60_000);

app.get('/orders/:id/status', (req, res) => {
  res.json({ status: jobStatus[req.params.id] ?? 'unknown' });
});

app.get('/orders', async (req, res) => {
  const limit = req.query.limit || 50;
  const { rows: orders } = await pool.query(
    `SELECT * FROM orders WHERE user_id = $1 ORDER BY created_at DESC LIMIT ${limit}`,
    [req.user.id],
  );
  for (const order of orders) {
    const tags = await pool.query('SELECT tag FROM order_tags WHERE order_id = $1', [order.id]);
    order.tags = tags.rows.map((t) => t.tag);
  }
  res.json(orders);
});

app.post('/admin/prices/:userId', async (req, res) => {
  const { model, credits } = req.body;
  await pool.query(
    `INSERT INTO price_lists (user_id, model, credits) VALUES ($1, $2, $3)
     ON CONFLICT (user_id, model) DO UPDATE SET credits = EXCLUDED.credits`,
    [req.params.userId, model, credits],
  );
  res.sendStatus(204);
});

app.use((err, req, res, next) => {
  console.error(err);
  res.status(500).json({ error: err.message, stack: err.stack });
});

process.on('uncaughtException', (err) => {
  console.error('uncaught', err);
});

app.listen(process.env.PORT || 3000, () => console.log('orders service up'));

const client = await pool.connect();
try {
  await client.query('BEGIN');
  const charged = await client.query(
    `UPDATE wallets SET balance = balance - $1
     WHERE user_id = $2 AND balance >= $1 RETURNING balance`,
    [cost, userId],
  );
  if (charged.rowCount === 0) {
    await client.query('ROLLBACK');
    return res.status(402).json({ error: 'insufficient credits' });
  }
  const { rows } = await client.query(
    `INSERT INTO orders (user_id, prompt, cost, status, idempotency_key)
     VALUES ($1, $2, $3, 'pending', $4) RETURNING id`,
    [userId, req.body.prompt, cost, req.header('idempotency-key')],
  );
  await client.query('COMMIT');
  // ...
} catch (err) {
  await client.query('ROLLBACK');
  throw err;
} finally {
  client.release();
}

WITH claimed AS (
  UPDATE orders SET status = 'refunded'
  WHERE id = $1 AND status IN ('pending', 'running', 'failed')
  RETURNING user_id, cost
)
UPDATE wallets w SET balance = w.balance + c.cost
FROM claimed c WHERE w.user_id = c.user_id
RETURNING w.balance;

async function callProvider(prompt, options, { signal, idempotencyKey }) {
  return retry(async () => {
    const res = await fetch(`${PROVIDER_URL}/v1/generate`, {
      method: 'POST',
      headers: { 'content-type': 'application/json', 'idempotency-key': idempotencyKey },
      body: JSON.stringify({ prompt, model: options.model, max_tokens: options.maxTokens }),
      signal: AbortSignal.any([signal, AbortSignal.timeout(30_000)]),
    });
    if (!res.ok) {
      throw new ProviderError(`provider ${res.status}`, {
        status: res.status,
        retryable: res.status === 429 || res.status >= 500,
      });
    }
    return res.json();
  }, { retries: 3, shouldRetry: (e) => e.name === 'TimeoutError' || e.retryable });
}

The flaky test

The same service, a week later. Work out where to look next at each step.

Part 5: Defend Your Design

In the second half of many interviews, you describe a system you built and the interviewer pushes on it. They are checking three things: that you know the mechanism (not just the name of the pattern), that you know its failure modes, and that you chose it over alternatives for reasons.

The answers below are generic and correct models of each mechanism. Adapt them to what you actually built. If your system differs (a different isolation level, a lease instead of a held lock), say what you did and why; never claim a detail you cannot explain. "We didn't handle that; here is how I would" is a strong answer. A confident wrong one is not.

Claim 1: "Our credit ledger is race-safe. The balance check and the charge are one SQL statement, and a refund can't be paid twice even though four recovery paths can reach the same failed job."

Follow-up questions:

1. Two requests for the same user arrive in the same millisecond. Walk me through what Postgres does.
2. Why one statement instead of SELECT … FOR UPDATE in a transaction, or a Redis lock?
3. What exactly makes the refund single-use? What is unique, or what is conditional?
4. Which isolation level do you run, and does the design depend on it?
5. How do you know the balance matches the ledger? What if it drifts?

Model answer. "The charge is UPDATE wallets SET balance = balance - $cost WHERE user_id = $u AND balance >= $cost RETURNING balance. The UPDATE takes a row lock. A second concurrent UPDATE on that row waits until the first commits. Under Read Committed, Postgres then re-evaluates the WHERE against the newly committed row, so the second request's check sees the balance after the first charge. Zero rows back means insufficient credits. The check and the write cannot be separated, so there is no window. A ledger entry is inserted in the same transaction.

For refunds, the decision is made by the database, not by the caller. Every recovery path calls one function, which does a conditional transition: UPDATE jobs SET refund_state = 'refunded' WHERE id = $1 AND refund_state = 'none' RETURNING …, and credits the wallet in the same transaction only if a row came back. The alternative is a unique constraint on the ledger, such as (job_id, kind), so a second refund insert fails. Either way, four paths can race; exactly one wins and the others are no-ops.

SELECT … FOR UPDATE works too, but it is two round trips and holds the lock while application code runs between them; I'd use it if I needed logic between the read and the write. A Redis lock adds a second system and has a TTL: if the holder pauses past the TTL, two holders run at once unless every write checks a fencing token. The database is already the source of truth, so it should enforce the rule.

It relies on Read Committed's re-check for single-row updates. Under Serializable, the second transaction would get a serialization failure instead, and we'd need retry logic. A periodic reconciliation compares each balance with the sum of its ledger entries and alerts on any difference."

Step through the four-paths problem with and without the conditional update:

Claim 2: "Our generation engine is crash-resumable. We checkpoint after every step."

Follow-up questions:

1. What is a step, and what exactly is in a checkpoint? Where is it stored?
2. The process dies after the step's side effect (the LLM call returned, you were billed) but before the checkpoint is written. What happens on resume?
3. How do you tell a crashed run from a slow one? Who resumes it?
4. Can two workers resume the same run at once? What stops them?
5. You deploy new code that changes the steps while runs are in progress. What happens?
6. Why not Temporal, Inngest, or a workflow engine?

Model answer. "A run is a row with a current step number, and each step's output is a row keyed by (run_id, step). A step reads the previous outputs, does its work, and then writes its output and advances the run in one transaction. On resume, a worker reads the last committed step and continues from there.

That makes it at-least-once per step, not exactly-once. If the process dies after the LLM returned but before the commit, the step runs again on resume. So each step must be safe to repeat: it uses an idempotency key like run_id:step where the provider supports it, and writes with ON CONFLICT DO NOTHING. For a paid call that cannot be deduplicated, we accept the occasional repeated cost; it is bounded by the number of crashes.

Crashed vs slow: a run being worked on has a lease, locked_by and locked_until, renewed by a heartbeat. A run whose lease has expired is up for grabs. Two workers: the checkpoint write is conditional on still holding the lease (WHERE run_id = $1 AND locked_by = $me), so a worker that paused past its lease and wakes up gets zero rows and stops. That is a fencing check.

Deploys: runs record the pipeline version they started with, and either the old step code is kept until those runs drain, or steps are designed so a newer version can read an older checkpoint.

A workflow engine gives all of this, plus timers and signals, but it is another system to run and learn. With a few linear steps and Postgres already there, a table was enough. I'd move to one when the workflows get branches, long waits, or human approval steps."

Claim 3: "LLM calls fail over across three providers, with circuit breakers, timeouts and a spend cap."

Follow-up questions:

1. What timeout do you use, and how did you pick it? Is it the same for streaming?
2. Which errors fail over, which are retried on the same provider, and which fail immediately?
3. When you fail over, what happens to the request still running at the first provider?
4. Describe the breaker's states. What closes it again? Is breaker state per instance or shared?
5. Is the spend cap race-safe across instances? What happens when it is hit mid-request?
6. Why not an LLM gateway that does this for you?

Model answer. "Each attempt gets its own AbortSignal.timeout, set from the provider's latency distribution with margin, and the whole request has an overall deadline so failover cannot add up to minutes. For streaming, there are two limits: time to first token, and maximum gap between chunks, because a stream that started can still stall.

Errors are classified. 429 and 5xx and timeouts are transient: retry with backoff, or fail over. 400-class errors, like a prompt that is too long or blocked by a content filter, fail the same way everywhere, so they fail immediately. Retrying them on three providers triples the cost of a guaranteed failure.

On failover, the first request is aborted with its AbortController, so we stop holding the connection and, for streaming, stop paying for tokens we will not use. Without that, a 'failed' provider can still finish and bill us.

Each provider has a breaker. Closed: calls go through, failures are counted in a rolling window. Open: after a threshold, calls skip that provider straight away. After a cooldown it goes half-open: it allows one probe request; success closes it, failure re-opens it. Without the half-open state, a breaker that opened never recovers. Our breakers are per instance: each instance learns within a few failures, and it avoids a shared dependency in the failure path. Shared state in Redis reacts faster across a fleet but adds a call to every request.

The spend cap is enforced atomically in the database: reserve an estimate before the call with a conditional update, SET spent = spent + $est WHERE spent + $est <= cap, then adjust to the real token count afterwards. A plain read-then-check would let several instances pass the check together. When the cap is hit, new requests are rejected; in-flight ones finish.

A gateway handles routing and keys well. We kept it in code because failover decisions depend on our own error classification and per-feature budgets, and it removed a network hop. I'd revisit that with more providers or teams."

Claim 4: "We run a Postgres job queue with FOR UPDATE SKIP LOCKED, behind an outbox dispatcher."

Follow-up questions:

1. What problem does the outbox solve? What would break without it?
2. Write the claim query. What does SKIP LOCKED do that FOR UPDATE alone does not?
3. A worker dies holding a job. How does the job get picked up again?
4. What delivery guarantee does this give, and what does that require of handlers?
5. What are the limits: throughput, table bloat, ordering?
6. Why not Redis/BullMQ, SQS or Kafka?

Model answer. "The outbox solves the dual write. If a request commits an order and then enqueues a job in a separate system, a crash between the two leaves a committed order with no job, or a job for an order that rolled back. Instead, the order and an outbox row are written in one transaction. A dispatcher reads committed outbox rows and turns them into jobs, marking each one dispatched. If it crashes, it re-reads the same rows.

The claim is:

WITH next AS (
  SELECT id FROM jobs
  WHERE status = 'queued' AND run_at <= now()
  ORDER BY run_at
  LIMIT 10
  FOR UPDATE SKIP LOCKED
)
UPDATE jobs SET status = 'running', locked_by = $1,
       locked_until = now() + interval '5 minutes', attempts = attempts + 1
FROM next WHERE jobs.id = next.id
RETURNING jobs.*;

FOR UPDATE alone makes the second worker wait for the first worker's rows, so workers queue behind each other. SKIP LOCKED makes it skip those rows and take the next ones, so workers claim different jobs without blocking.

We claim in a short transaction and then hold a lease, rather than holding the transaction open during the work. A long open transaction holds a connection and stops vacuum cleaning up dead rows. The cost is that a dead worker's job is not released automatically: a reaper requeues jobs whose locked_until has passed.

That gives at-least-once delivery: a worker can finish the work and die before marking it done. So handlers are idempotent, keyed by job ID, and the 'done' update is conditional on still holding the lease.

Limits: every claim and completion is a write, so a very hot queue table needs an index on (status, run_at), aggressive autovacuum, and deleting or archiving finished jobs. Ordering is best-effort across workers. Workers poll; LISTEN/NOTIFY can wake them sooner.

Against Redis or SQS: the queue is transactional with the business data, which removes the dual write for enqueueing, and it is one less system to run. A dedicated broker wins at much higher throughput, or for fan-out to many consumers, which is Kafka's job."

Claim 5: "The API contract is typed with oRPC and Zod, and shared as an npm package."

Follow-up questions:

1. What is in the package, and what stays in the server?
2. A mobile app version from three months ago is still installed. What happens when you change a response shape?
3. Which changes are breaking? How do you version the package?
4. Do you validate on the client too? What is the runtime cost?
5. Why not OpenAPI with generated clients, GraphQL, or tRPC?

Model answer. "The package holds the contract: the procedures, and Zod schemas for each input and output. The server imports it and implements each procedure; clients import it and get typed calls. It contains no server code and no server dependencies, so nothing from the backend leaks into the app bundle.

Types are compile-time only. They tell the client what to send; they do not stop anyone sending something else. So the server always validates input with the schema at the boundary. Validating outputs on the server catches us breaking our own contract.

Old mobile clients are the real constraint: they cannot be redeployed. So changes are additive. New fields are optional or have defaults; a field is never renamed or given a new meaning; removing one waits until the oldest supported app version no longer reads it. Anything else is a new procedure or a versioned one. The package follows semver, and CI type-checks the consumers against a change before it is published.

Compared with OpenAPI-generated clients: the same idea, but the schemas are the source of truth in TypeScript rather than in a YAML file, and an OpenAPI document can still be generated for outside consumers. tRPC infers the client from the server's router type, which works best in one monorepo; a published contract package works across repositories and separate release cycles. GraphQL solves a different problem, flexible querying across many clients, at the cost of a schema layer, caching complexity and query-cost limits we did not need."

Claim 6: "Our LLM pipeline is asynchronous on BullMQ and Redis, with retries and dead-letter queues."

Follow-up questions:

1. What delivery guarantee does BullMQ give? Can a job run twice?
2. Which failures are retried, and how? What stops a job retrying forever?
3. BullMQ has no built-in dead-letter queue. What is yours, exactly, and who looks at it?
4. Redis restarts or runs out of memory. What is lost?
5. The job is enqueued after a database write. What if the process dies in between?
6. You also run a Postgres queue elsewhere. Why Redis here?

Model answer. "BullMQ is at-least-once. A worker holds a lock on its active job and renews it in the background. If the lock is not renewed (the process died, or the event loop was blocked longer than the lock duration by CPU-heavy work) the job is detected as stalled and moved back to waiting, so another worker runs it again. So handlers are idempotent: results are written keyed by job ID, and paid calls carry an idempotency key where the provider supports it. CPU-heavy work goes to a worker thread or a sandboxed processor, so it does not block lock renewal.

Retries use attempts with exponential backoff. Errors that will never succeed, such as validation failures, throw BullMQ's UnrecoverableError so they fail without using up attempts. When the final attempt fails, the job lands in the failed set; our 'dead-letter queue' is a failed handler that, once attemptsMade reaches the limit, copies the job's data and error into a separate queue or table, alerts, and provides a replay command after a fix. A DLQ nobody reads is just a slower way of losing jobs, so the alert matters.

Redis must be configured for this: maxmemory-policy noeviction, because an evicting policy can silently delete queue keys, and persistence (AOF) set to the loss we can accept. Without AOF, a Redis restart loses jobs added since the last snapshot.

Enqueueing after a database commit is a dual write. Either enqueue from an outbox, or make the database the record of what should exist and run a reconciliation that re-enqueues anything without a result.

Redis here because the pipeline needed rate limiting per provider, many short jobs, and BullMQ's tooling, and Redis was already there for caching. The Postgres queue fits where jobs must be transactional with business data. The trade-off is one more stateful system to keep durable."

Part 6: Phrases That Signal Seniority

The Causal Chains

Review Checklist

1. Entry points: list every route, worker, timer and process.on. Anything that runs at import time?
2. Module-level state: every Map, object, array and let at the top of a file. Is it per-user or per-tenant? What happens with two instances?
3. External calls: timeout, retry policy, idempotency key, cancellation, for each one.
4. Read → await → write: every pair on the same data. Can it be one conditional statement or a transaction?
5. Money and state transitions: is every transition conditional on the current state? Can every recovery path call it safely twice?
6. Errors: typed errors with cause; no swallowed catch; async handlers wired to the framework's error path.
7. Floating promises: forEach(async, calls without await, .then without .catch.
8. Durability: does any work live only in setTimeout, an array, or an emitter listener?
9. Lifecycle: SIGTERM handling, server.close, timers cleared, uncaughtException exits.
10. Tests: is there a concurrent test for each money path? Are the clock, fetch and the database injectable?

What's Next

Chapter 11 — The Polyfill Set is the reference for the "build it from scratch" round: bind, the array methods, deep clone, memoize, EventEmitter, the promise combinators, debounce, asyncPool and retry, each with its edge cases. For more review practice, the gym's review-drills/ folder has larger files with an answer key.