Chapter 8 — The Node Process Under Failure (and Real Parallelism)

Crashes, shutdowns, a blocked loop, the threadpool, leaks, and the point where you need real threads

Prerequisites: Chapter 1 — The Event Loop Under Pressure, Chapter 6 — Cancellation and Cleanup, Chapter 7 — Controlling Concurrency. Runtime basics: FE-10. The debugging side of the same machine: Debugging 08 — Inside the Node Process (profiles, heap snapshots, --inspect). Practice: the gym, exercise 04-job-runner.

How to read this chapter

Chapters 1 to 7 treated the Node process as a thing that is always there. This chapter is about the process itself: what happens when an error escapes, when the orchestrator sends SIGTERM with requests and jobs still running, when one piece of code holds the thread for 300 ms, when a hidden pool of four threads is full, and when memory grows until the process is killed. The last part covers real parallelism, where two pieces of JavaScript really do run at the same instant, and the races you already know become data races on memory.

Every number in this chapter was measured on Node 22 on a laptop. Yours will differ; the shape will not.

Part 1: When an error escapes

Unhandled rejections in Node 22

setTimeout(() => console.log('timer still ran?'), 100);
async function chargeUser() { throw new Error('ledger write failed'); }
chargeUser(); // nobody awaits it
console.log('after call');

Output: after call, then the stack trace for ledger write failed, then exit code 1. The timer never prints.

The mechanism: chargeUser() returns a rejected promise and nothing attaches a handler. After the current macrotask and its microtasks finish, Node checks for promises that were rejected and still have no handler. The default mode since Node 15 is --unhandled-rejections=throw: Node emits 'unhandledRejection', and if no listener exists, it raises the reason as an uncaught exception. With no 'uncaughtException' listener either, the process exits with code 1 before the next timer phase.

Two consequences people miss:

• "Later" is too late. const p = Promise.reject(err); setTimeout(() => p.catch(...), 0) also crashes. The check runs after the microtask queue drains, before any timer. This is how for await over an array of promises crashes (Chapter 9) and why a pool must attach handlers to every task it starts.
• An 'uncaughtException' listener also receives unhandled rejections, with origin === 'unhandledRejection', if there is no 'unhandledRejection' listener. So one global "log it" handler silently disables both crash paths.

Why "log and continue" is wrong

process.on('uncaughtException', (err) => {
  logger.error({ err }, 'uncaught exception');
});

What's wrong?

process.on('uncaughtException', (e) => console.log('logged:', e.message));
const locks = new Set();
function handle(id) { locks.add(id); JSON.parse('{bad'); locks.delete(id); }
setTimeout(() => handle('wallet:42'), 0);
setTimeout(() => console.log('still running, locks held:', [...locks]), 50);

function crash(kind) {
  return (err) => {
    logger.fatal({ err, kind }, 'fatal, exiting');
    process.exitCode = 1;
    shutdown('fatal'); // the drain from Part 2, with its hard-exit timer
  };
}
process.on('uncaughtException', crash('uncaughtException'));
process.on('unhandledRejection', crash('unhandledRejection'));

Part 2: Graceful shutdown on SIGTERM

On a deploy, Kubernetes sends SIGTERM to the old pod, waits terminationGracePeriodSeconds (30 s by default), then sends SIGKILL. With no SIGTERM handler, Node exits at once with code 143 (128 + 15). Every in-flight request gets a reset connection, and every job mid-step stops wherever it was.

A correct shutdown does five things, in this order:

1. Stop taking new work. server.close() stops accepting connections. Health checks return 503 so the load balancer stops routing (endpoint removal is asynchronous, so a pod often still gets traffic for a second or two after SIGTERM). The job worker stops claiming jobs.
2. Drain in-flight requests, including keep-alive connections (see below).
3. Stop jobs at a safe point: abort their signal and let each one stop at its next step boundary, after its checkpoint.
4. Close resources: the DB pool, Redis, the queue client, flush logs and metrics. After the drain, because draining requests still need the pool.
5. Exit, with a hard-exit timer in case something never finishes. The timer is unref()ed so it does not itself keep the process alive.

The keep-alive trap

server.close() stops new connections and, since Node 19, closes connections that are idle at that moment. A keep-alive connection that is mid-request when close() runs finishes its response and then stays open, idle, until a keep-alive timeout closes it. Its close() callback waits for it. In a test, a fetch for a 300 ms request in flight at SIGTERM delayed exit to 3.3 s; adding a sweep that calls server.closeIdleConnections() every 100 ms brought it down to 0.38 s. A connection that was already idle at SIGTERM closed at once either way. Fetch, axios and most HTTP clients use keep-alive by default, so this happens on every deploy.

Build: a shutdown handler

Spec: an HTTP server plus an in-process job runner. On SIGTERM or SIGINT: stop accepting, mark health as draining, let in-flight requests finish, abort jobs so they stop at the next step boundary, then close the pool and exit 0. If draining takes longer than a grace period, exit 1. A second signal must not start a second shutdown.

import http from 'node:http';
import { setTimeout as delay } from 'node:timers/promises';

let shuttingDown = false;
const jobs = new Set(); // promises of in-flight jobs
const jobAbort = new AbortController();

const server = http.createServer(async (req, res) => {
  if (req.url === '/health') {
    res.statusCode = shuttingDown ? 503 : 200;
    return res.end(shuttingDown ? 'draining' : 'ok');
  }
  await handle(req, res);
});

async function runJob(job, signal) {
  for (let step = job.checkpoint; step < job.steps.length; step++) {
    if (signal.aborted) return; // stop between steps, never inside one
    await job.steps[step]();
    await saveCheckpoint(job.id, step + 1); // after the step's side effects
  }
}

function startJob(job) {
  const p = runJob(job, jobAbort.signal).finally(() => jobs.delete(p));
  jobs.add(p);
}

async function shutdown(reason) {
  if (shuttingDown) return;
  shuttingDown = true;
  logger.info({ reason }, 'draining');

  const hardExit = setTimeout(() => {
    logger.error('drain timed out, exiting hard');
    process.exit(1);
  }, 25_000); // below Kubernetes' 30 s grace period
  hardExit.unref();

  const closed = new Promise((resolve) => server.close(resolve));
  // close() only closes connections that are idle now. A keep-alive connection
  // that finishes its request later stays open, so sweep until all are gone.
  const sweep = setInterval(() => server.closeIdleConnections(), 100);
  sweep.unref();

  jobAbort.abort();
  await Promise.allSettled([closed, ...jobs]);
  await pool.end();
  logger.info('clean exit');
  process.exitCode = 0; // let the loop empty and exit on its own
}

process.once('SIGTERM', () => shutdown('SIGTERM'));
process.once('SIGINT', () => shutdown('SIGINT'));
server.listen(3000);

What happens to a job mid-step

A job is a sequence of steps with side effects: call the LLM, charge credits, write the result, send the webhook. Shutdown can arrive between any two lines. That is why a crash-resumable engine writes a checkpoint after each step, and why each step must be safe to run twice.

This is the same reasoning as the credit ledger: the refund cannot be paid twice because the database enforces it, not because the code path is unlikely to run twice.

Spot the bug: the shutdown handler

process.on('SIGTERM', async () => {
  logger.info('shutting down');
  server.close();
  await pool.end();
  await queue.close();
  process.exit(0);
});

What's wrong?

Part 3: Event loop lag and blocking

Node runs your JavaScript on one thread. While a function runs, no other request makes progress: not its timers, not its I/O callbacks, not /health. Chapter 1 showed this with timers. Here are the three blockers you meet in production:

Why the regex explodes: (\w+\s?)+ can split "aaaa" into groups in many ways (aaaa, aaa|a, aa|aa, a|a|a|a...), because the \s? is optional. When the final ! makes the match fail, the engine tries every split before giving up: about 2ⁿ of them. Measured: 20 letters 26 ms, 24 letters 78 ms, 26 letters 310 ms, 28 letters 1,256 ms. A user who types a long title into the admin panel takes down every request on that process. This is a ReDoS (regular expression denial of service).

Measuring it

import { monitorEventLoopDelay } from 'node:perf_hooks';

const h = monitorEventLoopDelay({ resolution: 10 });
h.enable();
setInterval(() => {
  const ms = (ns) => +(ns / 1e6).toFixed(1);
  metrics.gauge('event_loop_delay_p99_ms', ms(h.percentile(99)));
  metrics.gauge('event_loop_delay_max_ms', ms(h.max));
  h.reset();
}, 10_000).unref();

Two things to know when you read these numbers. The values include the sampling interval: an idle loop with resolution: 10 reads about 10 to 11 ms, so subtract the resolution. And a single long block shows in max, not in p99: in a 2-second test with one 120 ms JSON.parse, p99 was 11.5 ms and max was 128.7 ms. Alert on max (or p99.9) per interval.

The investigation below starts from a symptom you will recognise.

Part 4: The libuv threadpool

Not all of Node's "async" I/O is done by the OS's event mechanisms. Some operations have no non-blocking OS API, so libuv runs them on a threadpool of 4 threads and posts the result back to the event loop.

Four threads is not many. Eight concurrent crypto.pbkdf2 calls (eight logins at the same moment) and one tiny fs.readFile:

import { pbkdf2 } from 'node:crypto';
import { readFile } from 'node:fs/promises';

const t0 = performance.now();
const at = () => Math.round(performance.now() - t0);
const hash = (i) => new Promise((resolve) => pbkdf2('pw' + i, 'salt', 200_000, 64, 'sha512', () => resolve(at())));

const hashes = Promise.all(Array.from({ length: 8 }, (_, i) => hash(i)));
const file = readFile(new URL(import.meta.url)).then(() => at());
console.log('hashes done at', await hashes, 'ms');
console.log('fs.readFile done at', await file, 'ms');

With the default pool: four hashes finish at about 60 ms and the other four at about 115 ms, in two waves. The file read, a few hundred bytes, finishes at 117 ms, because it queued behind the hashes. With UV_THREADPOOL_SIZE=8, all eight hashes finish in one wave at about 100 ms.

This is why "the file reads are slow" or "DNS is slow" during a login spike or a burst of image compression: they are all in the same four-slot queue. The event loop itself shows no lag, which makes it confusing.

Fixes: set UV_THREADPOOL_SIZE (up to 1024) in the environment before the process starts. Setting process.env.UV_THREADPOOL_SIZE in code only works if it runs before anything has used the pool, which is fragile. Size it to the work, not to the core count alone. Also put a concurrency limit (Chapter 7) in front of heavy crypto or zlib work, and cache DNS or use keep-alive agents so hot paths do not call dns.lookup per request.

Part 5: What keeps the process alive, and what leaks

Timers and handles keep it alive

Node exits when the event loop has nothing left to wait for. An active setInterval, a listening server, an open socket, or a pending timer all count.

const flush = setInterval(flushMetrics, 1000);

A script with this line never exits on its own (tested: still running after 2.5 s with nothing else to do). In a CLI, a migration script or a test file, that is a hang. flush.unref() tells Node "do not stay alive just for this", and the same script exits as soon as its real work is done. Use unref() for background timers (metrics flush, cache sweeps, the hard-exit timer), and clear them in shutdown so the last flush actually runs.

Leaks from unbounded maps and listeners

A leak in a garbage-collected language is memory that is still reachable, so the GC must keep it. The usual reachable-forever places are module-level Maps and listeners on long-lived emitters.

const bus = new EventEmitter(); // lives as long as the process

app.get('/stream/:jobId', (req, res) => {
  bus.on('config-changed', () => res.write('event: reload\n\n'));
  // ...
});

Each request adds a listener and nothing removes it. The listener's closure holds res, so each finished response and its socket stay in memory. On the 11th listener Node prints MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 config-changed listeners added. That warning is the cheapest leak detector you have; do not silence it with setMaxListeners(0). The fix is to remove the listener when the request ends:

const onChange = () => res.write('event: reload\n\n');
bus.on('config-changed', onChange);
res.on('close', () => bus.off('config-changed', onChange));

The same shape: an in-memory rate limiter Map keyed by user id with no expiry, a promise cache with no eviction (Chapter 7), a Set of in-flight job ids that misses the delete on the error path. Use an LRU or TTL, delete in finally, and graph process.memoryUsage().heapUsed after GC. Debugging 08 shows how to find the retainer with heap snapshots.

Part 6: Real parallelism

Everything so far, and every race in Chapter 5, is concurrency: tasks taking turns on one thread, switching only at await and callback boundaries. Between two awaits your code cannot be interrupted. worker_threads removes that guarantee for memory you choose to share. Each worker is a separate V8 isolate with its own heap and event loop, running at the same time as the main thread on another core.

Message passing and its cost

By default workers share nothing. postMessage(value) makes a structured clone: a deep copy, serialised on the sender and deserialised on the receiver. Functions, class prototypes and sockets cannot be cloned (DataCloneError). Measured round trips:

Objects are expensive to copy, typed arrays are cheap to copy, and transferring an ArrayBuffer moves ownership instead of copying. That shapes how you offload work. Moving a 96 MB JSON.parse into a worker and posting all 400,000 parsed rows back blocked the main thread for about 150 ms while it deserialised them, longer than parsing on the main thread (120 ms). Posting back only a summary blocked it for about 1 ms. Do the heavy work and the reduction in the worker; send back small results or transferable buffers.

Starting a worker took about 11 ms and several MB in a test. Create a fixed pool of workers at startup (or use piscina) instead of one per request.

A real data race

A SharedArrayBuffer is memory that several threads see at once. Now counter[0]++ is a real race, because it is three steps (load, add, store) and another core can run its own three steps in between.

import { Worker, isMainThread, workerData } from 'node:worker_threads';

const N = 1_000_000;

async function run({ workers = 4, atomic = false } = {}) {
  const sab = new SharedArrayBuffer(4);
  const counter = new Int32Array(sab);
  await Promise.all(
    Array.from({ length: workers }, () => {
      const w = new Worker(new URL(import.meta.url), { workerData: { sab, atomic } });
      return new Promise((resolve, reject) => {
        w.once('error', reject);
        w.once('exit', resolve);
      });
    }),
  );
  return Atomics.load(counter, 0);
}

if (isMainThread) {
  console.log(await run());                 // expected 4,000,000
  console.log(await run({ atomic: true })); // 4,000,000
} else {
  const counter = new Int32Array(workerData.sab);
  if (workerData.atomic) {
    for (let i = 0; i < N; i++) Atomics.add(counter, 0, 1);
  } else {
    for (let i = 0; i < N; i++) counter[0]++; // load, add, store: three steps
  }
}

Four workers, a million increments each. Five runs (in the test and from the command line) of the plain version gave between 1,189,732 and 2,084,514. Every run of the Atomics.add version gave exactly 4,000,000. The file is both the main script and the worker script, which is why the isMainThread check decides which half runs.

This is where the section's core idea becomes literal. In single-threaded JS, a check-then-act is only a race across an await. With shared memory it is a race between any two instructions. Use Atomics for counters and flags, Atomics.wait / Atomics.notify to build a lock if you must, and prefer message passing so you do not share at all.

Workers vs a separate process vs a queue

Part 7: Cluster, PM2, and per-process state

node:cluster and PM2's cluster mode run N copies of your server, one per core, sharing one port. Each copy is a separate process with its own heap. Anything you keep in memory is per process:

import cluster from 'node:cluster';
import http from 'node:http';

if (cluster.isPrimary) {
  for (let i = 0; i < 4; i++) cluster.fork();
} else {
  const hits = new Map(); // "max 5 generations per user per minute"
  http.createServer((req, res) => {
    const n = (hits.get('user-42') ?? 0) + 1;
    hits.set('user-42', n);
    res.statusCode = n > 5 ? 429 : 200;
    res.end(`${process.pid} count=${n}`);
  }).listen(3999);
}

Twenty requests from the same user: all twenty returned 200. The connections were spread across four workers, each with its own Map, so each saw at most five. The limit is really 5 × the number of processes, and it changes when you scale.

The same applies to every in-memory structure from this section and Chapter 7:

Say it in the interview: "This lock is correct in one process. We run four pods with two cluster workers each, so it's eight independent locks. The real guard has to be in Postgres: a unique constraint or a conditional update."

Part 8: The Node event loop phases, in the sim

On the Node phases tab, step through process.nextTick, promise callbacks, setTimeout(0) and setImmediate inside an I/O callback, and predict the order before each step. It is the same machinery that decides when the unhandled-rejection check in Part 1 runs (after the microtasks) and why a blocked task in Part 3 delays every phase.

The Causal Chains

Review checklist

1. Global handlers: process.on('uncaughtException' | 'unhandledRejection') that log without exiting.
2. Fire-and-forget calls (doThing() with no await or .catch): each is a possible crash.
3. Signal handling: is there a SIGTERM handler; does it stop accepting, drain, sweep keep-alive connections, stop jobs at a step boundary, close resources after the drain, and have an unref'd hard-exit timer?
4. Job steps: a checkpoint after each step, and each step idempotent (keyed by job id and step).
5. Main-thread CPU: JSON.parse / JSON.stringify on unbounded payloads, *Sync crypto and fs in request paths, regexes with nested quantifiers on user input.
6. Event loop delay is exported as a metric (max per interval, not only p99).
7. Threadpool pressure: heavy crypto, zlib or fs work at high concurrency; UV_THREADPOOL_SIZE set in the environment if needed.
8. Timers: background setIntervals are unref()ed and cleared on shutdown.
9. Growth: module-level Map/Set/arrays without eviction; .on( without a matching .off(; setMaxListeners(0).
10. Per-process assumptions: in-memory locks, rate limits, caches and crons in a service that runs more than one process.

What's next

Large payloads came up three times in this chapter: the 96 MB JSON.parse, the rows posted back from a worker, and memory that grows with each request. Chapter 9 — Streams and Async Iterators is how you avoid holding them at all: backpressure, pipeline, Web Streams, and an LLM SSE stream parsed correctly across arbitrary chunk boundaries.