Chapter 7 — Controlling Concurrency

Pools, single-flight, batching, retries, breakers and rate shaping: deciding how much runs at once

Prerequisites: Chapter 3 — Promises in Real Code (Promise.all vs allSettled), Chapter 5 — Race Conditions (check-then-act across an await), Chapter 6 — Cancellation and Cleanup (AbortController). Runtime basics: FE-10. Practice: the gym, exercises 03-llm-client, 04-job-runner, 06-cache-layer, 10-polyfill-kata.

How to read this chapter

Chapters 3 to 6 were about getting async code correct. This one is about getting it to behave under load. Every tool here answers one question: how many of these things should be running at the same moment, and what happens when they fail? Promise.all answers "all of them, and keep going". A for … await loop answers "one". Most real code needs an answer in between, plus a policy for failure.

Read the parts in order. Each one ends with the code you would write in the interview. The builds have full reference solutions in <details> blocks; try each one before you open it. All of the code was run on Node 22.

Part 1: Why "all at once" fails

// Send a digest email to 20,000 users.
await Promise.all(users.map((u) => mailer.send(u.email, digest(u))));

The mechanism: users.map calls the arrow function synchronously for every element before Promise.all even runs. Each call runs until its first await, which is after it has started its network request. So 20,000 requests start inside one macrotask. Nothing in JS slows them down; the event loop is free to start as many as you ask for.

What breaks, roughly in this order:

The serial version, for (const u of users) await mailer.send(...), has none of these problems and takes 20,000 × 200 ms ≈ 67 minutes. The answer is a limit: run K at a time. With K = 20 the same job takes about 3.5 minutes and the provider sees a steady 100 requests per second instead of a spike.

Try it in the sim. On Pool limit, raise K until the API starts returning 429s, then find the K that finishes fastest without errors. On Serial vs parallel, compare the three strategies against the same API.

This is the same problem as the database. A Postgres connection pool is a concurrency limit around connections, and max_connections is the limit around the pool. Your JS limit should sit below the tightest limit downstream, or you just move the queue somewhere you cannot see it.

Part 2: The pool

There are two common shapes. Know both.

Workers pulling from a list. Start K async functions. Each loops: take the next index, run the task, store the result at that index, repeat until the list is empty.

let next = 0;
async function worker() {
  while (next < items.length) {
    const i = next++;               // claim an index
    results[i] = await fn(items[i]);
  }
}
await Promise.all(Array.from({ length: limit }, worker));

Two things make this correct, and you should say both out loud:

• next < items.length and next++ run with no await between them. In JS, nothing else can run between two lines unless there is an await (or a callback boundary) between them. So two workers can never claim the same index. Compare Chapter 5: a check-then-act is only a race when there is an await between the check and the act. Here there is not.
• Order is preserved because each result goes to results[i], not results.push(...). Tasks finish in any order; the array index does not care.

A queue in front of a function (how p-limit works). When calls arrive over time (from many HTTP requests, say) there is no list to hand to a pool. p-limit wraps each call instead: limit(fn) puts fn in a queue and returns a promise. A counter tracks how many are active. When one settles, the counter goes down and the next queued function starts.

export function pLimit(concurrency) {
  const queue = [];
  let active = 0;

  const run = async ({ fn, resolve, reject }) => {
    active++;
    try {
      resolve(await fn());
    } catch (err) {
      reject(err);
    } finally {
      active--; // runs before the caller's await resumes
      next();
    }
  };

  const next = () => {
    if (active < concurrency && queue.length > 0) run(queue.shift());
  };

  return (fn) =>
    new Promise((resolve, reject) => {
      queue.push({ fn, resolve, reject });
      next();
    });
}

// One limiter per process, shared by every request that calls the provider:
const limitLLM = pLimit(8);
app.post('/summarise', async (req, res) => {
  res.json(await limitLLM(() => llm.complete(req.body.text)));
});

queue.shift() is O(n) on a big array; the real p-limit uses a linked-list queue. That is a fine thing to mention, not a bug to fix in an interview.

The first three are per process. Four instances with pLimit(8) send 32 at once. Chapter 8 comes back to this.

Spot the bug: the pool that keeps going

A backfill re-embeds every document. It is meant to stop on the first error so someone can look at it.

async function backfill(docs) {
  const results = [];
  let next = 0;

  async function worker() {
    while (next < docs.length) {
      const i = next++;
      results[i] = await embedAndStore(docs[i]);
    }
  }

  await Promise.all([worker(), worker(), worker(), worker()]);
  return results;
}

backfill(allDocs).catch((err) => {
  log.error('backfill failed, stopping', err);
});

What's wrong?

const ac = new AbortController();
async function worker() {
  while (next < docs.length && !ac.signal.aborted) {
    const i = next++;
    try {
      results[i] = await embedAndStore(docs[i], { signal: ac.signal });
    } catch (err) {
      ac.abort(err);
      throw err;
    }
  }
}

Build: asyncPool(limit, items, fn)

Spec:

• Runs fn(item, index, signal) with at most limit calls in flight.
• Resolves to an array of results in input order.
• On the first rejection: rejects with that error, starts no new items, and aborts signal so in-flight calls can stop.
• Validates limit. Works with an empty list and with limit > items.length. Accepts any iterable.
• Variant asyncPoolSettled(limit, items, fn): never rejects; resolves to {status, value} / {status, reason} objects in input order, like Promise.allSettled.

export async function asyncPool(limit, items, fn) {
  if (!Number.isInteger(limit) || limit < 1) {
    throw new RangeError('limit must be an integer >= 1');
  }
  const list = Array.from(items);
  const results = new Array(list.length);
  const ac = new AbortController();
  let next = 0;

  async function worker() {
    // `next < list.length` and `next++` run with no await between them,
    // so two workers can never take the same index.
    while (next < list.length && !ac.signal.aborted) {
      const i = next++;
      try {
        results[i] = await fn(list[i], i, ac.signal);
      } catch (err) {
        ac.abort(err); // stop the other workers from taking new items
        throw err;
      }
    }
  }

  const workers = Array.from({ length: Math.min(limit, list.length) }, worker);
  await Promise.all(workers);
  return results;
}

export async function asyncPoolSettled(limit, items, fn) {
  if (!Number.isInteger(limit) || limit < 1) {
    throw new RangeError('limit must be an integer >= 1');
  }
  const list = Array.from(items);
  const results = new Array(list.length);
  let next = 0;

  async function worker() {
    while (next < list.length) {
      const i = next++;
      try {
        results[i] = { status: 'fulfilled', value: await fn(list[i], i) };
      } catch (reason) {
        results[i] = { status: 'rejected', reason };
      }
    }
  }

  await Promise.all(Array.from({ length: Math.min(limit, list.length) }, worker));
  return results;
}

Part 3: Semaphores

A pool limits tasks. A semaphore limits a section of code, wherever it is called from. Use it when the limited part is only one step of a longer function, for example "at most 3 concurrent calls to the image provider, but the rest of the request is unlimited".

export class Semaphore {
  #permits;
  #waiters = [];
  constructor(permits) {
    this.#permits = permits;
  }
  async acquire() {
    if (this.#permits > 0) {
      this.#permits--;
      return;
    }
    await new Promise((resolve) => this.#waiters.push(resolve));
    // the releaser handed its permit straight to us
  }
  release() {
    const wake = this.#waiters.shift();
    if (wake) wake();
    else this.#permits++;
  }
  async use(fn) {
    await this.acquire();
    try {
      return await fn();
    } finally {
      this.release();
    }
  }
}

Look at release(). It does not increment the counter and then wake a waiter. It hands the permit directly to the first waiter. If it incremented first, a brand new caller arriving before the waiter's microtask ran could see permits > 0 and take the permit, and the woken waiter would also proceed: two holders for one permit. Handing the permit over closes that gap. (The check permits > 0 and the decrement in acquire are safe for the same reason as next++ in the pool: no await between them.)

Always prefer use(fn) over bare acquire/release, because of the next drill.

Spot the bug: three failures and it hangs

const providerSlots = new Semaphore(3);

export async function generateImage(prompt) {
  await providerSlots.acquire();
  const image = await imageProvider.create({ prompt }); // throws on 503
  providerSlots.release();
  return image;
}

What's wrong?

export async function generateImage(prompt) {
  return providerSlots.use(() => imageProvider.create({ prompt }));
}

Part 4: Single-flight, and the rejected-promise trap

A popular cache key expires. In the next 50 ms, 300 requests miss it. Each one calls the database (or worse, the LLM to regenerate a summary). That is a cache stampede, and it is the same check-then-act race as Chapter 5: every request checks "is it cached?", sees no, and acts.

The trick is to store the promise, not the value. A promise exists the moment the load starts, so the second caller can find it before any data has come back. Then delete it when it settles, so the next call after that starts fresh.

Build: singleFlight(fn)

Spec: returns run(key, ...args). While a call for key is in flight, every run(key) returns the same promise. When it settles (either way), the entry is removed. A rejection is shared by the callers who were waiting, but the next call retries.

export function singleFlight(fn) {
  const inflight = new Map();
  return function run(key, ...args) {
    const existing = inflight.get(key);
    if (existing) return existing;
    const p = Promise.resolve()
      .then(() => fn(key, ...args))
      .finally(() => {
        // only delete our own entry, never a newer one
        if (inflight.get(key) === p) inflight.delete(key);
      });
    inflight.set(key, p);
    return p;
  };
}

// Usage: in front of a cache miss
const loadSummary = singleFlight((docId) => db.summaryFor(docId));
async function getSummary(docId) {
  const hit = await redis.get(`summary:${docId}`);
  if (hit) return JSON.parse(hit);
  const value = await loadSummary(docId);
  await redis.set(`summary:${docId}`, JSON.stringify(value), 'EX', 300);
  return value;
}

Spot the bug: the cache that never recovers

const userCache = new Map();

export function getUser(id) {
  if (!userCache.has(id)) {
    userCache.set(id, db.fetchUser(id));
  }
  return userCache.get(id);
}

During a 30-second database blip, some user lookups time out. The database recovers. An hour later, those same users still get errors on every request, and only a restart fixes it. What's wrong?

export function getUser(id) {
  let p = userCache.get(id);
  if (!p) {
    p = db.fetchUser(id).catch((err) => {
      if (userCache.get(id) === p) userCache.delete(id); // forget failures
      throw err;
    });
    userCache.set(id, p);
  }
  return p;
}

Part 5: Batching and DataLoader-style coalescing

Single-flight merges identical requests. Batching merges different ones. The classic case is the N+1 query: a GraphQL resolver or an admin-panel list calls getUser(id) once per row, and 50 rows means 50 queries. A loader collects every load(id) made in the same tick and sends one WHERE id = ANY($1).

export function createLoader(batchFn) {
  let queue = [];
  let scheduled = false;

  function dispatch() {
    const batch = queue;
    queue = [];
    scheduled = false;
    const keys = [...new Set(batch.map((b) => b.key))];
    Promise.resolve()
      .then(() => batchFn(keys))
      .then((values) => {
        if (!Array.isArray(values) || values.length !== keys.length) {
          throw new Error('batchFn must return one value per key, in order');
        }
        const byKey = new Map(keys.map((k, i) => [k, values[i]]));
        for (const b of batch) {
          const v = byKey.get(b.key);
          v instanceof Error ? b.reject(v) : b.resolve(v);
        }
      })
      .catch((err) => {
        for (const b of batch) b.reject(err);
      });
  }

  return function load(key) {
    return new Promise((resolve, reject) => {
      queue.push({ key, resolve, reject });
      if (!scheduled) {
        scheduled = true;
        queueMicrotask(dispatch);
      }
    });
  };
}

const userLoader = createLoader(async (ids) => {
  const { rows } = await pool.query('SELECT * FROM users WHERE id = ANY($1)', [ids]);
  const byId = new Map(rows.map((r) => [r.id, r]));
  return ids.map((id) => byId.get(id) ?? new Error(`user ${id} not found`));
});

The mechanism is the microtask queue. The first load() in a tick schedules dispatch as a microtask. Every other load() that runs before that microtask (all the synchronous calls from a .map, for example) just joins the queue. Then dispatch runs once with all the keys. The real DataLoader waits slightly longer than one microtask (in Node it chains a resolved promise into process.nextTick) so that loads made from several just-resolved promises still land in the same batch.

The rules the batch function must follow, and that reviewers check:

• Same length and order as the keys. SQL does not return rows in ANY($1) order, and missing rows are simply absent. Map by id, as above.
• Per-item errors go in the array as Error objects, so one missing user does not fail the other 49.
• Scope the loader per request if it caches. A module-level loader with a cache serves user A's data to user B, and never sees updates.
• Set a max batch size; ANY($1) with 50,000 ids is its own incident.

Part 6: Retry with backoff, jitter and a budget

Which errors are retryable

A retry is a bet that the same request might succeed if sent again, and that sending it again is safe. Both halves matter.

Idempotency is the second half. A timeout tells you nothing about whether the server did the work. If POST /charges times out, the charge may have happened; retrying it may charge twice. Retrying is only safe when the operation is naturally idempotent (GET, PUT of a full value, an LLM completion with no side effects) or you send an idempotency key the server uses to deduplicate. This is the same reason the credit ledger makes refunds idempotent: four recovery paths can reach the same job, and a retry is just a fifth.

Exponential backoff with full jitter

Exponential backoff doubles the ceiling each attempt: 100 ms, 200, 400, 800, capped at some max. Full jitter picks the actual wait uniformly between 0 and that ceiling:

delay = Math.random() * Math.min(max, base * 2 ** attempt);

Why the randomness? Imagine 500 clients whose requests all failed at 12:00:00.000 because the provider had a hiccup. With a fixed 1-second delay, all 500 retry at 12:00:01.000, the provider falls over again, and they all retry together at 12:00:02.000. They stay synchronised forever. Backoff without jitter only spaces the waves out; it does not break them up. Full jitter spreads each wave across the whole window, so the provider sees a trickle.

In the sim's Retry storm tab, run fixed-delay retries and watch the waves line up, then switch to backoff with jitter.

Retry budgets: retries multiply

Retries at several layers multiply. If the provider SDK retries 3 times (4 attempts), your LLM client wraps it with 4 attempts, and the route handler wraps that with 4 more, one user request can become 4 × 4 × 4 = 64 provider calls while the provider is down. That is exactly when it needs less traffic. Two rules:

1. Retry at one layer, usually the one closest to the failing call, and turn the others off.
2. Use a retry budget: retries may be at most a fixed share (say 10%) of recent requests. When the budget is spent, fail fast. Under a full outage, traffic grows by 10%, not by 4×.

class RetryBudget {
  #requests = 0;
  #retries = 0;
  constructor({ ratio = 0.1, minPerWindow = 10, windowMs = 10_000 } = {}) {
    this.ratio = ratio;
    this.minPerWindow = minPerWindow;
    setInterval(() => { this.#requests = 0; this.#retries = 0; }, windowMs).unref();
  }
  onRequest() { this.#requests++; }
  canRetry() {
    const allowed = Math.max(this.minPerWindow, this.#requests * this.ratio);
    if (this.#retries >= allowed) return false;
    this.#retries++;
    return true;
  }
}

const budget = new RetryBudget();
// every outbound call: budget.onRequest(); then
retry(callProvider, { shouldRetry: (err) => isRetryable(err) && budget.canRetry() });

With every call failing, 1,000 requests with up to 3 retries each produce 1,100 provider calls instead of 4,000.

Spot the bug: the retry helper

const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

export async function withRetry(fn) {
  for (let i = 0; i < 4; i++) {
    try {
      return await fn();
    } catch (err) {
      await sleep(1000);
    }
  }
}

// used as:
const charge = await withRetry(() => payments.post('/charges', { userId, amount }));

What's wrong? There are at least four bugs.

const charge = await retry(
  () => payments.post('/charges', { userId, amount }, { headers: { 'Idempotency-Key': jobId } }),
  { retries: 3, base: 200, max: 5_000, signal: AbortSignal.timeout(15_000) },
);

Build: retry(fn, { retries, base, max, signal })

Spec:

• Calls fn({ attempt, signal }). On success, returns the value.
• On a retryable error, waits random() * min(max, base * 2 ** attempt) and tries again, up to retries extra attempts (so retries + 1 calls in total). Honours a retryAfterMs on the error if it is longer.
• On a non-retryable error or when out of retries, rethrows the last error.
• If signal aborts (before an attempt, during an attempt, or during the sleep), stops at once and rejects with signal.reason.

export function sleep(ms, signal) {
  return new Promise((resolve, reject) => {
    if (signal?.aborted) return reject(signal.reason);
    const onAbort = () => {
      clearTimeout(timer);
      reject(signal.reason);
    };
    const timer = setTimeout(() => {
      signal?.removeEventListener('abort', onAbort);
      resolve();
    }, ms);
    signal?.addEventListener('abort', onAbort, { once: true });
  });
}

export function backoffDelay(attempt, base, max, random = Math.random) {
  const cap = Math.min(max, base * 2 ** attempt);
  return random() * cap; // full jitter: anywhere in [0, cap)
}

export function isRetryable(err) {
  if (err?.name === 'AbortError') return false; // the caller cancelled
  if (err?.name === 'TimeoutError') return true; // a per-attempt timeout
  const status = err?.status;
  if (status === 429 || status === 502 || status === 503 || status === 504) return true;
  if (typeof status === 'number') return false; // 400, 401, 404, 409, 422...
  return ['ECONNRESET', 'ETIMEDOUT', 'ECONNREFUSED', 'EAI_AGAIN'].includes(err?.code);
}

export async function retry(fn, { retries = 3, base = 100, max = 10_000, signal, shouldRetry = isRetryable } = {}) {
  for (let attempt = 0; ; attempt++) {
    signal?.throwIfAborted();
    try {
      return await fn({ attempt, signal });
    } catch (err) {
      if (signal?.aborted) throw signal.reason; // the deadline, not this attempt
      if (attempt >= retries || !shouldRetry(err)) throw err;
      let delay = backoffDelay(attempt, base, max);
      if (err?.retryAfterMs != null) delay = Math.max(delay, err.retryAfterMs);
      await sleep(delay, signal);
    }
  }
}

Part 7: Timeouts per attempt vs an overall deadline

They answer different questions:

Combine them with AbortSignal.any (Node 20.3+ and modern browsers):

const deadline = AbortSignal.timeout(10_000); // the whole operation

const reply = await retry(
  ({ signal }) => {
    const attempt = AbortSignal.any([signal, AbortSignal.timeout(3_000)]);
    return fetch(PROVIDER_URL, { method: 'POST', body, signal: attempt });
  },
  { retries: 3, base: 200, max: 2_000, signal: deadline },
);

The attempt signal fires on whichever comes first: the 3-second attempt timeout or the 10-second deadline. When the attempt timeout fires, the error is a TimeoutError, retry sees that deadline is not aborted, and retries. When the deadline fires, retry sees signal.aborted and stops. A test with a 250 ms deadline and 50 ms attempts that always hang gives five attempts and rejects with TimeoutError at about 250 ms.

Propagate the deadline downward. If the HTTP request has 10 s and you have already spent 7 s, the job you enqueue or the provider call you make should get 3 s, not a fresh 10 s. Pass the signal (or the absolute deadline) through, the same way you pass a trace ID.

Part 8: The circuit breaker

Retries help with brief failures. When a provider is down for minutes, retries just add load and make every user wait for the full timeout before failing. A circuit breaker notices the dependency is failing and fails fast for a while.

• Closed: calls go through. Count consecutive failures (or a failure rate over a window).
• Open: calls are rejected immediately with CircuitOpenError, without touching the provider. Your failover moves to the next provider.
• Half-open: after the cooldown, let one (or a few) trial calls through. Success closes the circuit; failure opens it again with a fresh cooldown.

Two bugs show up in almost every hand-written breaker. Gym exercise 03-llm-client has the first one.

Bug 1: it never half-opens. The usual causes:

• The "rejected while open" path calls trip() again, which resets openedAt = now. Under steady traffic, every rejected call pushes the cooldown forward, so it never ends. A test with one request every 100 ms and a 1-second cooldown shows zero calls reaching the provider for as long as traffic continues. It only recovers at night, when traffic stops.
• The transition to half-open is driven by a setTimeout that gets cleared or never set, instead of by comparing the clock on the next call.
• The breaker only moves to half-open inside the success path, which can never run while open.

Bug 2: half-open lets the herd through.

A test of the unlimited version with 200 concurrent calls after the cooldown sends all 200 to the provider. The reference below sends 1.

Build: a CircuitBreaker class

Spec:

• new CircuitBreaker({ failureThreshold, cooldownMs, halfOpenMax, isFailure, now }), call(fn), and a readable state.
• Opens after failureThreshold consecutive failures. While open, rejects with CircuitOpenError without calling fn, and rejected calls do not extend the cooldown.
• After cooldownMs, the next call moves it to half-open. At most halfOpenMax trial calls run at once; others are rejected.
• A trial success closes it; a trial failure re-opens it with a new cooldown.
• isFailure(err) decides what counts (a 400 from the provider says nothing about its health).
• A result from a call that started in an earlier state must not move the current state (a slow call from before the trip that succeeds later should not close an open breaker).
• now is injectable for tests.

export class CircuitOpenError extends Error {
  constructor() {
    super('circuit open');
    this.name = 'CircuitOpenError';
  }
}

export class CircuitBreaker {
  state = 'closed';
  #failures = 0;
  #openedAt = 0;
  #trials = 0; // trial calls in flight while half-open
  #generation = 0; // bumps on every state change

  constructor({ failureThreshold = 5, cooldownMs = 30_000, halfOpenMax = 1, isFailure = () => true, now = Date.now } = {}) {
    Object.assign(this, { failureThreshold, cooldownMs, halfOpenMax, isFailure, now });
  }

  #moveTo(state) {
    this.state = state;
    this.#generation++;
    this.#failures = 0;
    this.#trials = 0;
    if (state === 'open') this.#openedAt = this.now();
  }

  async call(fn) {
    if (this.state === 'open') {
      // The clock decides, not a timer, and a rejected call does NOT reset openedAt.
      if (this.now() - this.#openedAt < this.cooldownMs) throw new CircuitOpenError();
      this.#moveTo('half-open');
    }
    if (this.state === 'half-open') {
      if (this.#trials >= this.halfOpenMax) throw new CircuitOpenError();
      this.#trials++;
    }

    const gen = this.#generation;
    const wasTrial = this.state === 'half-open';
    try {
      const result = await fn();
      if (gen === this.#generation) {
        if (wasTrial) this.#moveTo('closed');
        else this.#failures = 0;
      }
      return result;
    } catch (err) {
      if (gen === this.#generation && this.isFailure(err)) {
        if (wasTrial) this.#moveTo('open');
        else if (++this.#failures >= this.failureThreshold) this.#moveTo('open');
      } else if (gen === this.#generation && wasTrial) {
        this.#trials--; // not a provider failure (e.g. a 400): free the trial slot
      }
      throw err;
    }
  }
}

// One breaker per provider, around a call that has its own timeout:
const breakers = { openai: new CircuitBreaker(), anthropic: new CircuitBreaker(), gemini: new CircuitBreaker() };
const callProvider = (name, req) =>
  breakers[name].call(() => clients[name].complete(req, { signal: AbortSignal.timeout(20_000) }));

Part 9: Hedged requests and provider failover

Failover: if provider A fails, try B, then C. Hedging: if A has not answered within, say, its p95 latency, also start B and use whichever answers first. Both share one rule: when one attempt wins, abort the others. Otherwise the loser keeps running, holds a socket, and for an LLM keeps generating tokens you pay for.

Spot the bug: the timeout that doesn't stop anything

function withTimeout(promise, ms) {
  return Promise.race([
    promise,
    new Promise((_, reject) => setTimeout(() => reject(new Error('timeout')), ms)),
  ]);
}

export async function complete(prompt) {
  try {
    return await withTimeout(openai.stream(prompt), 8_000);
  } catch {
    return await withTimeout(anthropic.stream(prompt), 8_000);
  }
}

What's wrong?

export async function complete(prompt, { signal } = {}) {
  return failover(
    [
      (s) => openai.stream(prompt, { signal: AbortSignal.any([s, AbortSignal.timeout(8_000)]) }),
      (s) => anthropic.stream(prompt, { signal: AbortSignal.any([s, AbortSignal.timeout(8_000)]) }),
    ],
    { signal },
  );
}

Here is a failover that does both failover and hedging, and aborts every loser but never the winner (aborting the winner would kill a stream you are still reading).

export function failover(providers, { hedgeAfterMs = Infinity, signal } = {}) {
  return new Promise((resolve, reject) => {
    if (signal?.aborted) return reject(signal.reason);
    const controllers = [];
    const errors = [];
    let next = 0;
    let running = 0;
    let settled = false;
    let timer;

    const finish = (winner) => {
      settled = true;
      clearTimeout(timer);
      signal?.removeEventListener('abort', onAbort);
      controllers.forEach((ac, i) => {
        if (i !== winner) ac.abort(new DOMException('lost the race', 'AbortError'));
      });
    };
    const onAbort = () => {
      if (settled) return;
      finish(-1);
      reject(signal.reason);
    };
    signal?.addEventListener('abort', onAbort, { once: true });

    const launch = () => {
      if (settled || next >= providers.length) return;
      const i = next++;
      const ac = new AbortController();
      controllers[i] = ac;
      running++;
      clearTimeout(timer);
      if (next < providers.length && Number.isFinite(hedgeAfterMs)) {
        timer = setTimeout(launch, hedgeAfterMs);
      }
      Promise.resolve()
        .then(() => providers[i](ac.signal))
        .then(
          (value) => {
            if (settled) return;
            finish(i);
            resolve(value);
          },
          (err) => {
            running--;
            if (settled) return;
            errors[i] = err;
            if (next < providers.length) launch(); // fail over now, don't wait for the timer
            else if (running === 0) {
              finish(-1);
              reject(new AggregateError(errors, 'all providers failed'));
            }
          },
        );
    };
    launch();
  });
}

Tested behaviour: a primary that fails fast moves to the secondary immediately (not after hedgeAfterMs); a slow primary gets a hedge after hedgeAfterMs and is aborted when the hedge wins; all failing rejects with an AggregateError holding every error; the caller's signal aborts every attempt.

Hedging costs money. For cheap idempotent reads (a cache, a search replica) hedging at p95 adds about 5% load and cuts tail latency a lot. For LLM calls, a hedge can double your token bill. Hedge on time to first token rather than full completion, or not at all, and put each provider behind its own breaker so a dead provider is skipped instead of tried.

Part 10: Debounce and throttle

These shape input events, not network calls, but they are concurrency control all the same: they decide how many handler runs a burst of events produces.

• Debounce: wait until the events stop for wait ms, then run once with the latest arguments. Each new event restarts the timer.
• Throttle: run at most once per wait ms. Leading means the first event in a window runs immediately. Trailing means the last event in a window runs when the window ends.

Try the three tabs. On Debounce, type a burst and watch only the last event fire. On Throttle, see the steady rhythm. On Leading vs trailing, turn each edge off and watch which events get dropped.

Spot the bug: the debounce that doesn't

function SearchBox({ onResults }) {
  const [query, setQuery] = useState('');

  const search = async (q) => {
    const res = await fetch(`/api/search?q=${encodeURIComponent(q)}`);
    onResults(await res.json());
  };

  return (
    <input
      value={query}
      onChange={(e) => {
        setQuery(e.target.value);
        debounce(search, 300)(e.target.value);
      }}
    />
  );
}

Typing "llm" sends three requests. What's wrong?

const debouncedSearch = useMemo(() => debounce((q) => searchRef.current(q), 300), []);
useEffect(() => () => debouncedSearch.cancel(), [debouncedSearch]); // no call after unmount

Build: debounce(fn, wait) with cancel and flush

Spec: the debounced function stores the latest arguments and this, and restarts a wait ms timer on every call. When the timer fires, fn runs once with the latest arguments. cancel() drops the pending call. flush() runs the pending call now (and returns its result); it does nothing if nothing is pending. pending() reports whether a call is scheduled.

export function debounce(fn, wait) {
  let timer = null;
  let lastArgs;
  let lastThis;

  function invoke() {
    const args = lastArgs;
    const ctx = lastThis;
    timer = null;
    lastArgs = lastThis = undefined;
    return fn.apply(ctx, args);
  }

  function debounced(...args) {
    lastArgs = args;
    lastThis = this;
    clearTimeout(timer);
    timer = setTimeout(invoke, wait);
  }
  debounced.cancel = () => {
    clearTimeout(timer);
    timer = null;
    lastArgs = lastThis = undefined;
  };
  debounced.flush = () => {
    if (timer === null) return undefined;
    clearTimeout(timer);
    return invoke();
  };
  debounced.pending = () => timer !== null;
  return debounced;
}

Build: throttle(fn, wait, { leading, trailing })

Spec: at most one call per wait ms. With leading (default true), the first call in a quiet period runs at once. With trailing (default true), calls made during a window are remembered and the latest one runs when the window ends; that trailing call opens a new window, so calls stay at least wait apart. With both false, nothing ever runs. Add cancel() and flush().

For events at 0, 20, 40, 60, 80, 120, 140 and 400 ms with wait = 100:

export function throttle(fn, wait, { leading = true, trailing = true } = {}) {
  let timer = null; // non-null while a window is open
  let pendingArgs = null;
  let pendingThis;

  function invoke(ctx, args) {
    fn.apply(ctx, args);
    timer = setTimeout(windowEnd, wait); // every call opens a fresh window
  }

  function windowEnd() {
    timer = null;
    if (trailing && pendingArgs) {
      const args = pendingArgs;
      const ctx = pendingThis;
      pendingArgs = pendingThis = null;
      invoke(ctx, args);
    }
  }

  function throttled(...args) {
    if (timer === null) {
      if (leading) {
        invoke(this, args);
        return;
      }
      timer = setTimeout(windowEnd, wait); // open a window without calling
    }
    if (trailing) {
      pendingArgs = args;
      pendingThis = this;
    }
  }
  throttled.cancel = () => {
    clearTimeout(timer);
    timer = null;
    pendingArgs = pendingThis = null;
  };
  throttled.flush = () => {
    if (!pendingArgs) return;
    clearTimeout(timer);
    const args = pendingArgs;
    const ctx = pendingThis;
    pendingArgs = pendingThis = null;
    invoke(ctx, args);
  };
  return throttled;
}

The Causal Chains

Review checklist

1. Promise.all(items.map(...)) over unbounded input (anything from a DB query, a file, a user upload). Ask: how big can it get, and what is the tightest limit downstream?
2. Every acquire(), active++, or lock has its release in a finally.
3. Pools stop on failure when they claim to: a flag checked before each new item, and a signal passed to in-flight tasks.
4. Promise caches (Map of promises): deleted on rejection, bounded in size, keyed by everything that changes the answer.
5. Retry helpers: rethrow the last error, check retryability, use jitter, honour Retry-After, and are abortable. Count retry layers along one call path.
6. Retried writes carry an idempotency key.
7. Every outbound call has a per-attempt timeout, and the request or job has an overall deadline that is passed down.
8. Circuit breakers: what moves them to half-open (a clock check, not a timer), how many trial calls pass, whether rejected calls touch openedAt, whether 4xx counts as failure.
9. Promise.race with a timeout or a hedge: is the loser aborted?
10. debounce / throttle created once (module scope, useMemo, useRef), cancelled on unmount, flushed before navigation when the last call matters.

What's next

Everything in this chapter assumes the process stays up. Chapter 8 — The Node Process Under Failure (and Real Parallelism) covers what happens when it does not: unhandled rejections, graceful shutdown with in-flight work, a blocked event loop, the libuv threadpool, and the point where you need real threads.