Chapter 1 — Timeouts and Deadlines

Every call needs a limit on how long it waits, and every request needs one budget that all its hops share

Prerequisites: SD QA Q33 — Timeout, deadline, retry budget (the three ideas side by side), JS 7, Part 7 — Timeouts per attempt vs an overall deadline (AbortSignal.any in code), Debugging 13 — Slow, flaky and falling over (Case 3: timeouts in the wrong order, Case 7: one slow dependency took everything down), Debugging 9 — Inside the database (Part 6: Postgres timeouts).

How to read this chapter

The deep chapters above already explain why timeouts and deadlines exist. This chapter covers what they leave thin: which part of a call each timeout measures, how a deadline is actually carried from one service to the next, and what happens, step by step, when there is no timeout at all. Every number in Parts 2, 3 and 4 was measured on Node 22.23 against local servers. The code is plain ESM and runs as written.

The interviewer's version of this topic is short: "That downstream now takes 30 seconds. What happens?" A strong answer traces the wait through the design and names the resource that runs out first. The parts below give you the mechanism for that answer.

Part 1: One call has six phases, and each can hang

"Add a timeout" hides a question: a timeout on what? An outbound HTTP call from a Node service goes through these phases, in order:

Two consequences matter in the room.

First, a failure in phases 1–4 means the request never reached the server's application code. A connect timeout on a POST /charges is safe to retry: no charge happened. A timeout in phase 5 or 6 is different: the server may have done the work. Chapter 2 builds on this. One caveat: behind a load balancer, your connect timeout only measures the hop to the load balancer. Its own connection to the backend is a separate hop with its own timeout, and it reports that failure to you as a 502 or 504.

Second, the phases fail for different reasons, so they need different values. Inside one region a TCP connect takes about a millisecond. A connect that has not finished in a second is not "slow"; the host is gone. A response, on the other hand, can legitimately take 800 ms. One shared 10-second number is too long for the first and possibly too short for the second.

Idle timeouts mean two different things

The word idle is used for two unrelated timers, and mixing them up is a common interview slip.

The keep-alive rule from that case is worth one sentence in an interview: the side that reuses connections must give up on idle ones first. If the load balancer keeps idle connections for 60 s, the app's keepAliveTimeout must be longer than 60 s.

A streaming LLM call needs three timeouts

The reader's own systems stream LLM tokens, which makes the distinction concrete. One total timeout cannot tell a healthy long answer from a stuck one. A streaming call needs:

The values are examples; derive yours from the provider's measured latency (Part 5).

Part 2: What Node 22 actually does

These are the defaults you get if you write await fetch(url) and nothing else. Each row was run against a local server: one that accepts the connection and never answers, one that sends a byte per second forever, a plain TCP server that never speaks TLS, and the unroutable address 10.255.255.1, which drops packets the way a dead host behind a firewall does.

What to take from the table:

• Node's fetch (undici) has defaults, and they are long: a 10 s connect timeout, then 300 s for the response headers and 300 s of idle time between body chunks. "No timeout" in Node fetch means "five minutes per stuck call", which under load is the same as forever.
• node:http has no connect timeout of its own. The operating system gives up: 75 s on macOS, about two minutes on Linux with default settings (net.ipv4.tcp_syn_retries = 6).
• The timeout option of http.request only emits an event. It is an idle timer. Unless your handler calls req.destroy(), the request stays open. This is one of the most common "we had a timeout" bugs in Node code review.
• Idle timeouts don't cap total time. A server that sends one byte per second resets bodyTimeout forever. Only a total timeout (AbortSignal.timeout) stops it, and it works during the body read too, not only until headers arrive.
• undici's own timers are coarse, about half a second late in every run. That is fine for safety limits. Use AbortSignal.timeout for anything you want precise.

A Node outbound client that covers every phase looks like this:

import { Agent } from 'undici'; // the npm package; Node's built-in fetch uses its own copy

// One Agent per dependency (see Part 3). Short connect, coarse safety limits.
const pricingPool = new Agent({
  connections: 20,              // per origin; the default is unlimited
  connect: { timeout: 1_000 },  // TCP + TLS
  headersTimeout: 2_000,
  bodyTimeout: 2_000,           // idle gap, not total
});

export async function getPrice(sku, { signal } = {}) {
  // Total cap for this call, and the request's deadline, whichever is first.
  const attempt = AbortSignal.any([signal, AbortSignal.timeout(500)].filter(Boolean));
  const res = await fetch(`https://pricing.internal/prices/${sku}`, { dispatcher: pricingPool, signal: attempt });
  if (!res.ok) throw Object.assign(new Error(`pricing ${res.status}`), { status: res.status });
  return res.json(); // the signal still covers the body read
}

One more measured detail: a total timeout started before the pool wait includes the pool wait. In Part 3's run, a /price call with a 2 s AbortSignal.timeout failed with TimeoutError after 2,003 ms without ever leaving the process, because all 10 connections to that host were held by hung calls. The error looked exactly like a slow server. Log the time spent waiting for a connection separately, or you will blame the wrong service.

The server side has timeouts too

Your service is someone else's downstream. Defaults, measured on Node 22 and Postgres 18:

Nothing in Node stops your handler from running for an hour. That is your code's job, and Part 4 is how.

Part 3: No timeout, as a cascade

Here is the failure the interviewer wants you to trace. The physics is Little's Law: in flight = rate × duration. At 50 requests per second and 80 ms each, 4 requests are in flight. If the duration becomes "forever", the in-flight count grows by 50 every second, and every one of them holds something: memory, a socket, a pooled connection, maybe a database connection.

Slow is worse than down. A dead server refuses connections or fails health checks; callers get fast errors and the load balancer removes it. A slow server passes health checks and keeps getting its share of traffic. In the sim, server B becomes 5× slower at 1 s but stays healthy. Predict what happens to p99 under round robin, then watch B's queue.

The measured run

The setup: an API with two routes. /home calls a recommendations endpoint and a price endpoint in parallel. /checkout calls only the price endpoint. Both go to the same downstream host through one shared undici Agent with 20 connections. At t = 0 the recommendations endpoint stops answering; the price endpoint stays healthy (5 ms). Load: 50 /home and 10 /checkout per second for 10 s. The client gives /checkout 5 s.

Read it as a timeline:

Three points to say out loud:

1. The downstream was not overloaded. It had 20 open requests the whole time. The failure was entirely waiting, inside the caller. Its database and CPU graphs look fine, which is why this failure confuses people during an incident.
2. The healthy endpoint died first in user terms. /checkout failed because it shared a queue with a call that never returned. The general version: a shared, bounded resource (connection pool, worker pool, thread pool, DB pool) plus an unbounded wait equals a full resource.
3. The timeout and the bulkhead fix different things. The timeout bounds how long anything waits (15 in flight instead of 480). The bulkhead bounds who shares the waiting. Chapter 3 covers bulkheads in depth.

Where else unbounded waits hide

The last measured detail: with the unlimited default dispatcher, 500 hung requests became 500 TCP connections on the downstream. An unbounded client pool does not protect you, and it moves the problem to the dependency, which now has to hold 500 connections it cannot serve.

Part 4: Deadlines that travel

A deadline is one budget for the whole request. Deadline propagation means every hop knows how much of it is left. Q33 explains why; here is how it is carried and what each hop does with it.

Relative, not absolute, on the wire

You can send either an absolute deadline ("finish by 14:02:07.250") or the time remaining ("you have 1,450 ms"). An absolute timestamp only works if both machines' clocks agree; a 300 ms clock skew silently moves the deadline by 300 ms. So the usual choice is to send the remaining time and convert it to a local deadline on arrival:

• gRPC does exactly this. The caller's deadline is sent as a grpc-timeout header holding a relative value (for example 1450m, meaning 1,450 milliseconds). The receiving server turns it into a local deadline, and a call that runs past it fails with status DEADLINE_EXCEEDED. In Go the deadline rides in the context and is passed on when you use that context for the next call. In Node's @grpc/grpc-js you pass a deadline in each call's options.
• Envoy sets x-envoy-expected-rq-timeout-ms on requests it forwards, telling the upstream the timeout Envoy will apply, so the service can give up early.
• Plain HTTP has no standard header. Pick one (x-request-timeout-ms below) and treat it like a trace ID: every outbound call carries it.

Relative time has one flaw: the time the request spent on the wire is not subtracted, so each hop's local deadline is slightly later than the caller's. Each hop should subtract a small reserve for that and for writing its own reply.

What each hop does

At every hop, the same five steps:

1. On arrival, compute a local deadline: now + header - reserve. If there is no header, use your own default for this route.
2. Before each outbound call, compute remaining = deadline - now. If it is less than what the call needs to be useful (say, the callee's p50), don't make the call: fail fast or use the fallback. Sending work that cannot finish only adds load.
3. Set the call's timeout to min(remaining, this call's own cap) and send remaining in the header.
4. Stop when the caller leaves. If the upstream aborts, its connection closes; in Node, res.on('close') fires with res.writableEnded === false. Abort your downstream calls and queries.
5. Pass the signal into everything that waits: fetch, the DB query, the LLM SDK call, the sleep between retries.

In Node it is a small middleware:

const RESERVE_MS = 50;

export function deadline(defaultMs) {
  return (req, res, next) => {
    const header = Number(req.get('x-request-timeout-ms'));
    const budget = Number.isFinite(header) && header > 0 ? Math.floor(header) : defaultMs;
    const at = Date.now() + budget - RESERVE_MS;

    const callerLeft = new AbortController();
    res.on('close', () => { if (!res.writableEnded) callerLeft.abort(new Error('caller left')); });

    req.deadline = {
      at,
      signal: AbortSignal.any([callerLeft.signal, AbortSignal.timeout(Math.max(0, at - Date.now()))]), // ms are integers
      remaining: () => at - Date.now(),
      // Options for one outbound call: its own cap, the shared deadline, and the header for the next hop.
      forCall(capMs) {
        const left = Math.floor(at - Date.now());
        if (left <= 0) throw Object.assign(new Error('deadline exceeded'), { status: 504 });
        const ms = Math.min(left, capMs);
        return {
          signal: AbortSignal.any([this.signal, AbortSignal.timeout(ms)]),
          headers: { 'x-request-timeout-ms': String(left) },
        };
      },
    };
    next();
  };
}

// usage
app.get('/cart', deadline(1_800), async (req, res) => {
  const opts = req.deadline.forCall(500);
  const price = await fetch(PRICING_URL, opts).then((r) => r.json());
  res.json({ price });
});

For the database, set the limit per transaction from the same budget: SET LOCAL statement_timeout = '<remaining> ms' at the start of the transaction (it resets at commit or rollback). For a queued job, store the absolute deadline on the job row; a worker that picks up an expired job drops it instead of running it. Chapter 4 covers that queue-timeout idea in depth.

Measured: fixed timeouts vs a propagated budget

A three-hop chain: A (200 ms of work) calls B (300 ms of work), which calls C. C needs 3 s. The user waits 2 s.

In the fixed version, every layer gave up only on its own wait. C did all its work after the user had been told it failed. If that work was a write and the client retries, it runs twice (Chapter 2). In the propagated version, C was told it had 1,448 ms when it started, and it stopped when B hung up.

Part 5: Choosing the numbers

Work top-down from what the user will wait

Start from the user-facing target, then split it. Each inner timeout must be shorter than its caller's, so the innermost work gives up first and its error reaches someone still listening (Debugging 13, Case 3 shows what happens when they grow inwards).

browser 10 s  >  edge / LB 3 s  >  API route deadline 2 s
                                     ├─ pricing call   500 ms   (critical)
                                     ├─ recs call      300 ms   (optional, has a fallback)
                                     └─ DB statement   800 ms   (SET LOCAL from the remaining budget)

Set each per-call timeout from the callee's measured latency

• Measure from the caller's side. The caller's p99 includes the network, TLS and the pool wait; the callee's dashboard does not.
• A timeout at p99 fails 1% of normal requests. That is too tight for most calls; a timeout should fire for broken, not for slow but normal. A common choice is a little above p99.9, or 2–3× p99, capped by the remaining deadline. Example: pricing p50 40 ms, p99 180 ms, p99.9 310 ms → 500 ms.
• Connect timeouts are short. Inside a region a connect takes about 1 ms, so 1–3 s is generous; across the internet, 3–5 s. A lost SYN is retransmitted after about a second on Linux, so a connect timeout below that allows no second try.
• Optional dependencies get tight timeouts and a fallback. Recommendations at 300 ms with "render without them" costs one strip; at 5 s it costs the page.
• Fan-out makes the tail worse. If a request waits for 5 parallel calls and each is slower than its p99 1% of the time, the request is slow whenever any of them is: 1 − 0.99⁵ ≈ 4.9%. Your page's p95 is roughly your dependencies' p99.
• Retries need room inside the deadline. With a 2 s budget and a 500 ms per-attempt timeout, at most three attempts fit, minus backoff. Retry only if remaining exceeds the expected attempt time (Chapter 2).
• Revisit the values. Latency drifts. Alert on the timeout rate per dependency; a rising rate means the value or the dependency changed.

When the Interviewer Breaks It

Try each one out loud before opening the answer.

Break 1: "Your payment provider now takes 30 seconds to respond."

Break 2: "Every call has a 2-second timeout, but some users wait 12 seconds."

Break 3: "The browser gave up after 2 seconds. Is your server still doing the work?"

Break 4: "Why 500 ms? Defend the number."

Break 5: "After a failover, the old database IP drops all packets. What do your services see?"

The Causal Chains

Checklist: add this to any design

1. A deadline per route, derived from what the user will wait, passed to every downstream call as remaining time.
2. A connect timeout (1–3 s in-region) separate from the response timeout (from the callee's p99.9, capped by the remaining deadline).
3. A total timeout that covers the body, plus an idle timeout for streams. Three for LLM streams: first token, between tokens, total.
4. Cancellation downward: one signal into fetch, DB queries, SDK calls and retry sleeps, aborted when the caller disconnects.
5. Database limits: statement_timeout (per role, and SET LOCAL from the budget), lock_timeout on migrations, idle_in_transaction_session_timeout, a pool acquire timeout.
6. A bounded client pool per dependency, so a hung dependency fills its own pool, not a shared one.
7. Fallbacks for optional calls, with tighter timeouts than critical ones.
8. Inner timeouts shorter than outer ones, all the way from the browser to statement_timeout.
9. Metrics: timeout rate per dependency, pool wait time, in-flight count. Alert on the in-flight count. It rises before anything else does.

What's next

A timeout tells you to stop waiting. It doesn't tell you whether the work happened. Chapter 2 — Retries and Idempotency covers what to do next: which failures are safe to retry, why a timeout means an unknown outcome, and how idempotency keys and deduplication make a second attempt harmless.