Chapter 8 — Inside the Node Process

Breakpoints on the Server, the Event Loop, CPU Profiles, Memory, and Bugs With No Error

Part 2: Instruments. Prerequisites: Chapters 1–7. Background: FE-10 — JavaScript Runtime Internals for the event loop, FE-07 for heap snapshots.

Opening

Some bugs have no error message. Every endpoint becomes slow at the same moment and recovers a minute later. Memory climbs for six hours until the container is killed. A request simply never returns. Health checks fail even though nothing crashed. Logs show nothing wrong, because nothing threw.

These are bugs in how the process is running, not in what one function returned. To find them you need to look inside the running Node process: pause it, profile it, measure its event loop, and inspect its memory. This chapter covers those instruments, and the mental model that makes their output make sense.

Part 1: The Model — One Thread for Your JavaScript

Node runs all your JavaScript on one thread. The event loop takes the next piece of work — a request handler, a resolved promise, a timer callback — runs it to completion, then takes the next one. While one piece of JavaScript is running, nothing else runs: no other request, no timer, no health check.

Work that would block — file system calls, DNS lookups with dns.lookup, some crypto, compression — is handed to libuv's thread pool, which has 4 threads by default (UV_THREADPOOL_SIZE). Network I/O does not use the pool; the operating system handles it asynchronously.

That model predicts the shape of Node's performance bugs:

Part 2: The Debugger on the Server

The same Sources panel from Chapter 5 works on Node.

node --inspect server.js         # start with the inspector on 127.0.0.1:9229
node --inspect-brk server.js     # same, but pause on the first line

Then open chrome://inspect in Chrome and click inspect under your process, or attach from VS Code (a "launch" or "attach" configuration). Every breakpoint type from Chapter 5 — conditional breakpoints, logpoints, pause on caught exceptions — works here, and the Scope pane shows the real request object, the real database result, the real config.

• Attach to a process that is already running by sending it SIGUSR1 (kill -USR1 <pid> on Linux/macOS). Node starts the inspector without restarting — useful when a bug only shows after hours of uptime.
• Pausing stops the whole process. On a shared server, a breakpoint freezes every request. Prefer logpoints anywhere other people's traffic flows.

Never expose the inspector port to a network. Anyone who can reach port 9229 can run arbitrary code inside your process. Keep it on 127.0.0.1 and reach it through an SSH tunnel or kubectl port-forward if you must.

Part 3: Event-Loop Blocking — the Bug That Slows Everyone

Common causes of a blocked event loop

Measuring it

You do not need to guess whether the loop is blocked. Measure the event-loop delay — how late timers fire compared with when they should:

import { monitorEventLoopDelay } from 'node:perf_hooks'

const h = monitorEventLoopDelay({ resolution: 20 })
h.enable()
setInterval(() => {
  // values are in nanoseconds
  console.log({ p99_ms: h.percentile(99) / 1e6, max_ms: h.max / 1e6 })
  h.reset()
}, 10_000)

A healthy server has a p99 delay of a few milliseconds. Spikes to hundreds of milliseconds or seconds, lined up with latency spikes on every endpoint, prove the loop was blocked. Export this as a metric (Chapter 10).

The fixes, in order of preference

1. Use the async version of the API (moves work to the pool or the OS).
2. Stream instead of loading everything — process rows and files in chunks.
3. Split the work into smaller pieces so other work can run between them.
4. Move CPU-heavy work off the main thread — a worker_threads worker, or a separate background job.

Part 4: CPU Profiles and Flame Graphs — Where Did the Time Go?

When the process is slow or CPU is high, do not guess which function is expensive. Record a CPU profile: the runtime samples the call stack thousands of times per second and records where time was spent.

node --cpu-prof server.js          # writes a .cpuprofile file when the process exits

Or, with the inspector attached, use the Performance (or Profiler) panel to record a few seconds while you reproduce the slowness. Open a .cpuprofile by dragging it into Chrome DevTools' Performance panel or into VS Code.

Reading a flame graph

A flame graph (or flame chart) draws the call stack:

• Each bar is a function. The bars below it are what called it; the bars above it are what it called.
• Width is time. A wide bar is a function that was on the stack for a long time.
• Look for wide bars near the top — functions that spent a lot of time doing their own work, not calling others. Those are your hot spots.
• Bottom-Up (or "self time") view sorts functions by the time spent in the function itself — the fastest way to find the hot spot.

A typical finding: a wide JSON.stringify bar under your exportCsv handler, or a wide regular-expression bar under a validation function. You now know exactly which line to fix.

Part 5: Memory — Growth, Leaks and the Two Kinds of Out-of-Memory

Reading process.memoryUsage()

Normal versus leaking

Healthy memory is a sawtooth: it rises as objects are created and drops when the garbage collector runs. A leak is a sawtooth whose floor keeps rising — after each collection, more memory survives than before.

The two endings from Chapter 3:

• heapUsed hits V8's limit → FATAL ERROR: Reached heap limit … JavaScript heap out of memory → exit code 134.
• rss hits the container limit (possibly with plenty of heap left, because buffers and native memory grew) → killed by the OOM killer → exit code 137 / OOMKilled.

Raising --max-old-space-size only moves the first limit. If it is set above the container's limit, you turn a clean 134 into a sudden 137.

Finding a leak

Use the same heap-snapshot workflow as FE-07: take a snapshot, run the suspected operation many times, take another, and compare — sort by objects whose count grew, then read their retainer chain (what is keeping them alive).

Ways to take a snapshot of a server process:

node --heapsnapshot-signal=SIGUSR2 server.js    # then: kill -USR2 <pid>
node --heapsnapshot-near-heap-limit=2 server.js # write up to 2 snapshots as the heap nears its limit

or call v8.writeHeapSnapshot() from code. A snapshot pauses the process and can need memory similar to the heap itself — on a container already near its limit, taking one can trigger the OOM kill. Take them on a copy of the service, or before memory is critical.

The usual server-side leaks

The MaxListenersExceededWarning is a gift. It is Node telling you, at the exact moment it happens, that listeners are piling up on one emitter. Run with --trace-warnings to get the stack trace of the line that added the eleventh listener.

Part 6: Async Bugs That Produce No Error

The promise that never settles. A request hangs until the client or proxy times out (often surfacing as a 504), and the server logs nothing — because nothing failed. Causes: a callback API wrapped in a Promise that never calls resolve on one branch; an outgoing call with no timeout to a service that never answers; waiting on a lock or a queue that never frees. The fix is twofold: find the missing branch, and put a timeout on every outbound call so a hang becomes an error you can see.

const res = await fetch(url, { signal: AbortSignal.timeout(5_000) })

The missing await. The function returns before the work is done. The work then fails later, outside any try/catch (Chapter 2), or the response is sent before the data is saved.

Unbounded concurrency. await Promise.all(tenThousandItems.map(processItem)) starts ten thousand operations at once — ten thousand database queries against a pool of ten connections, or ten thousand calls to an API with a rate limit. The symptom is pool timeouts, 429s, or memory spikes. Limit concurrency (process in batches, or use a small concurrency limiter).

"Why won't my process exit?" Something is keeping the event loop alive: an open server, a database pool, a setInterval, a socket. These are called open handles. process.getActiveResourcesInfo() lists their types, and test runners have options to report them (Jest: --detectOpenHandles). In tests, the fix is to close the pool and server in teardown.

Part 7: Diagnostic Flags and Built-in Tools

The diagnostic report is underused. When a process dies and you have nothing else, the report has the JavaScript stack, native stack, memory figures and libuv handle list from the moment of death.

Part 8: Graceful Shutdown — Part of Debugging Deploy-Time Errors

Many "random" errors cluster around deploys: a burst of 502s, ECONNRESETs, half-written jobs. The cause is usually a process that dies mid-request. When the orchestrator wants to stop a process, it sends SIGTERM, waits a grace period, then sends SIGKILL (exit 137 or 143 from Chapter 3).

A graceful shutdown handler uses that grace period:

process.on('SIGTERM', () => {
  server.close(async () => {       // stop accepting new connections; wait for in-flight requests
    await db.end()                 // close the pool
    process.exit(0)
  })
  server.closeIdleConnections()    // drop idle keep-alive sockets so close() can finish
  setTimeout(() => process.exit(1), 10_000).unref()  // give up before the orchestrator's SIGKILL
})

Pair it with a readiness check that starts failing as soon as shutdown begins, so the load balancer stops sending new traffic before the process stops accepting it.

The Worked Example — "Everything Is Slow Every Few Minutes"

Symptom. Several times an hour, every endpoint's p99 latency jumps to 4–5 seconds for about a minute, then recovers. There are no errors in the logs. CPU on one instance at a time spikes to 100%.

1. Fit the model. Every endpoint slow at once, on one instance at a time, with no errors — that is the signature of a blocked event loop (Part 1).

2. Measure. The event-loop delay metric (Part 3) shows p99 delay jumping from 3 ms to 4,100 ms at exactly the moments of the latency spikes. Confirmed: the loop is blocked.

3. Find what blocks it. Attach the inspector to one instance and record a CPU profile across a spike. The flame graph has one enormous bar: JSON.stringify, called from exportOrdersCsv.

4. Follow the trail. Logs filtered to route = "GET /admin/orders/export" show the endpoint is called at each spike time. It loads 200,000 orders into memory, builds one giant object, and stringifies it — about four seconds of synchronous CPU, during which no other request on that instance can run.

5. Fix. Stream the export: read rows with a cursor and write CSV lines to the response as they arrive, so no single step blocks for long. Longer term, make large exports a background job that emails a download link.

6. Prevent. Alert on event-loop delay p99, and add the export endpoint to a load test.

The Causal Chains

The Rule You Keep Forever

In Node, one slow synchronous function is everyone's latency. When the process misbehaves without an error, stop reading code and measure the runtime: event-loop delay for blocking, a CPU profile for where time goes, heap snapshots for what stays alive.

Vocabulary Map

Practice Problems

Problem 1 — Read the symptom. For each, say whether it points to event-loop blocking, thread-pool saturation, a leak, or a hanging promise: (a) one endpoint takes exactly 30 s and then the proxy returns 504, with no error logged; (b) all endpoints slow down together for 2 s every minute; (c) password hashing slows down sharply above 20 concurrent logins while CPU stays at 40%; (d) memory rises 50 MB per hour, every hour.

Problem 2 — The profile. A CPU profile of a slow search endpoint shows 70% of time in a function called matchesFilter, inside which almost all time is in RegExp.prototype.test. What do you suspect, and how would you confirm it?

Problem 3 — 134 or 137? A service is killed with OOMKilled. heapUsed never exceeded 300 MB, but rss reached the container's 1 GB limit, and arrayBuffers was 550 MB. Where would you look, and why would raising --max-old-space-size not help?

Problem 4 — Concurrency. A nightly job does await Promise.all(users.map(sendDigestEmail)) for 40,000 users, and the email provider starts returning 429. Rewrite the approach in words so the job finishes reliably.

Problem 5 — Deploy noise. Every deploy produces about 200 502 errors in one minute. Describe what the process is doing wrong during shutdown, and the two changes (one in the app, one in the load-balancer setup) that remove the errors.

What's Coming Next

A large share of "the server is slow" turns out to be "the server is waiting for the database." Chapter 9 looks at that wait from both sides: the connection pool in your app, and the database's own view of what is running, what is waiting, and what is blocking whom — including the migration that takes the whole site down by queueing behind one slow query.