Chapter 15 — Incidents, Postmortems, and the Story

Running an Incident, Learning From It, and Telling It in an Interview

Part 4: After the Bug. Prerequisites: Chapters 1–14B.

Opening

Every chapter so far has been about one engineer and one bug. In a production incident several people work at once, users are hurt while you think, and someone keeps asking "when will it be fixed?" The method from Chapter 1 still applies, but inside a team, under time pressure.

This chapter covers what happens around the bug: running an incident, learning from it, and telling the story in an interview. The hard part of an incident is rarely the bug. It is staying organised while the bug is still unknown.

Part 1: The Incident Lifecycle

An incident is an unplanned event that hurts users or puts them at risk and needs a coordinated response now. A broken, rarely used settings page is a ticket. Checkout failing for one customer in eight is an incident.

Every incident moves through the same stages:

Mitigated means users are no longer hurt; the cause may still be in the code. Resolved means the incident is over. The permanent fix often ships days later, as an action item.

When to declare

Declaring costs a channel and a few messages. Not declaring costs three engineers debugging in three private chats, with nobody telling support or deciding anything. Good teams lower the bar. Declare when:

• An alert on user impact fires (error rate, latency, an SLO burning fast — Chapter 10).
• There is any doubt about data: loss, corruption, data shown to the wrong user, a possible security problem.
• You have worked on something for 15–30 minutes and it is not getting better.
• More than one person or team is needed.

If it turns out to be nothing, close it with a note. Criticise people for declaring small incidents and they stop declaring — and the next big one starts late.

Severity levels

A severity level (SEV) is a shared label for how bad an incident is. It decides who is paged, how often updates go out, and who must be told. Every company defines its own — some start at SEV0, some use P1–P5 — but the shape is usually close to this:

Set severity fast and change it freely. Judge it by user impact, not by how frightening the cause looks: a boring typo that stops all payments is a SEV1; a scary warning with no user impact is not.

Measuring the response

• Time to detect (TTD) — from impact starting to a human knowing. Tests your monitoring.
• Time to acknowledge (TTA) — from the alert to someone accepting the page. Tests on-call.
• Time to mitigate (TTM) — from impact starting to impact stopping. Usually what matters most to users.
• Time to resolve (TTR) — from impact starting to the incident being declared over.

Averaged across incidents these become MTTD, MTTM and MTTR. Be careful with MTTR: the R is used for repair, recovery, restore, resolve and sometimes respond, which measure different things — ask what started and stopped the clock. Averages also hide shape (Chapter 10): one eight-hour incident and nine short ones average to an hour that describes none of them.

Part 2: Roles and Rules

Incidents go wrong predictably: everyone debugs and nobody coordinates, two people try two fixes at once, nobody tells support. Roles prevent this.

In a small incident one person may hold two roles, but never commander and main debugger: once a second person joins, the IC hands the debugging over.

The rules

• One incident channel, named after the incident (#inc-2026-10-10-checkout-errors). DMs are invisible to the scribe and to whoever joins next.
• A running timeline in UTC, because responders sit in different time zones and logs are usually stored in UTC.
• Decisions written down. "Decision: roll back checkout-api to v411. Owner: Priya." A decision only in someone's head gets remade or contradicted ten minutes later.
• Announce actions before taking them. "About to restart pod checkout-api-7d9f." Someone may say "wait — save its logs first."
• Explicit hand-offs. The outgoing person writes the state, open questions and anything in flight; the incoming one confirms "I am now IC."

The status update

Updates are for people outside the incident. A good one takes ten seconds to read:

Impact now:       who is affected, and how — in user terms
What we know:     facts so far, not guesses presented as facts
What we're doing: the current action
Next update:      HH:MM UTC

Example:

[SEV2] Checkout errors — update 1 — 12:08 UTC

Impact now: About 1 in 8 checkout attempts fails with an error. Browsing,
search and accounts work normally. Customers who see the error are not charged.
What we know: Errors started at 12:02 UTC, right after a release of the
checkout service.
What we're doing: Rolling back that release; expected to finish around 12:12 UTC.
Next update: 12:20 UTC, or sooner if anything changes.

No stack traces, no guessed root cause, no promise that the rollback will work. "Not charged" is there because support will be asked it first — and is written only because responders checked it. If you do not know something important, say so.

Send the next update when you promised, even with no news. "No change; still rolling back; next update 12:35 UTC" is a good update. Silence brings people into the incident channel to ask, pulling responders off the work.

Escalation and paging etiquette

• Page the role, not the person you know — use the owning team's on-call schedule.
• Make the page self-contained: "SEV2: POST /checkout 12% 5xx since 12:02 UTC, after checkout-api v412. Need someone who knows the pricing change. Channel: #inc-…." The person you wake should be useful in one minute.
• Escalate on a clock, not a feeling. No acknowledgement within the policy window (often 5–15 minutes): page the secondary. Stuck longer than agreed: pull in more help. Escalating is not failure.
• Wake people for a SEV1 without hesitating; think twice for a SEV3 at 3 a.m.

Part 3: The Mitigation Playbook

When users are hurt, stopping the harm comes before understanding the cause (Chapter 1). The usual levers, roughly from fastest and safest to slowest and riskiest:

Background: SD QA Q31 — Rate limiting, Q32 — Backpressure, Q30 — Circuit breakers.

Preserve evidence before destructive steps

Restarts, rollbacks, killed sessions and data fixes destroy the state that explains the bug: a restarted process loses the heap that was growing, a deleted pod its local logs. Spend a minute — not ten — keeping what you will need:

• The process: a heap snapshot or a diagnostic report (process.report.writeReport()) from one affected instance (Chapter 8). A snapshot pauses the process and needs extra memory, so take it from an instance out of traffic.
• One instance out of rotation: drain it — remove it from the load balancer but keep it running — instead of killing every bad instance.
• Local logs from the failing instance, if they are not shipped centrally.
• The database's view: copy pg_stat_activity and the blocking-PID query output (Chapter 9) before pg_terminate_backend.
• The bad data: copy affected rows to a separate table or file before a data-fix UPDATE or DELETE. Your fix might be wrong.

Note in the timeline what you captured and where. For a SEV1, a quick capture then mitigation; for memory creeping up over hours, take your time.

Rollback is not always safe

• Irreversible migrations. If the release dropped a column or rewrote data, the old code may fail on the new schema. Expand-and-contract migrations keep every step compatible with old and new code (PostgreSQL Ch 18).
• Version skew. Old and new versions run side by side during any rollout or rollback (Chapter 11, Case 6). Has the new version written rows, cache entries or messages the old one cannot read?
• Config is not in the deploy. Rolling back code does not undo a flag flip or an environment change made at the same time.
• Correlation is not causation. A cheap rollback is also a test: if errors do not drop, the release probably was not the cause.

Play it: you are the incident commander

The same incident, as the channel saw it

Part 4: The Postmortem

A postmortem (or incident review) is a written analysis of what happened, why, how the response went, and what will change. It turns an expensive incident into an improvement. Without it, the incident happens again.

Why blameless

A blameless postmortem assumes people acted reasonably given what they knew, and looks for causes in the system — code, tools, processes, incentives. This is about accurate information, not niceness: blamed people stop sharing what they saw and did. The engineer who ran the wrong command knows why it looked right, and that "why" is where the fix is.

"Human error" is never a root cause. People make mistakes at a steady rate. When one causes an incident, ask:

1. What made the mistake easy? Production and staging consoles that look identical; a command with no confirmation.
2. What made its consequences large? No review, no gradual rollout, no alert, no limit on what one command can delete.

Each answer is something you can change. "Be more careful" is not.

The template

"Where we got lucky" is the most underrated section: the same incident at 3 a.m. on a holiday would have been far worse. Each piece of luck is a risk to remove.

The 5 whys, and their limit

The 5 whys asks "why?" repeatedly until you reach something you can change. For the checkout incident in Chapter 9's worked example:

1. Why did checkouts fail? Requests timed out waiting for a database connection.
2. Why were there none free? All 40 were held by checkout transactions.
3. Why so long? Each transaction stayed open during the payment provider call.
4. Why was that call slow? The provider's latency rose from 400 ms to 25 s.
5. Why did a slow provider take down checkout? Our code put a network call inside a database transaction.

It pushed past "the provider was slow" to something in our control. But it produces one chain, and incidents rarely have one cause. This chain misses why nobody was alerted until users were failing, and why some customers were charged twice — contributing factors where the cheapest improvements often are. Use it to get past the first answer, not as a ritual.

Good action items

A good action item is specific (someone else could do it from the description), owned (one named person), verifiable (you can tell when it is done) and dated. "Be more careful with transactions", "improve monitoring" and "investigate timeouts" fail all four. "Warn on checkout-api pool waitingCount > 0 for 2 minutes, routed to checkout on-call; owner K. Menon; due 5 October" passes. A few finished items beat a long list nobody completes.

A complete worked postmortem

The incident from Chapter 9's worked example, written up in full. Names and numbers are illustrative.

Postmortem: Checkout failures during payment-provider slowdown — 2026-09-14

Severity: SEV1 (declared SEV2 at 14:04, raised at 14:12). IC: R. Iyer. Authors: R. Iyer, A. Rao.

Summary. From 13:55 to 14:38 UTC, 31% of checkout attempts failed with 504, and other checkout-api endpoints slowed (p99 from 300 ms to 9 s). Our payment provider slowed from about 400 ms to 25 s per call, and checkout held a database transaction open during that call, exhausting the connection pool. We mitigated at 14:27 by lowering the payment timeout and raising the pool size. The permanent fix moves the call out of the transaction and adds idempotency keys.

Impact. 43 minutes. About 4,500 of 14,600 checkout attempts failed; about 1,900 of those orders were not completed within 24 hours. GET /orders and GET /account p99 reached 9 s. 37 customers were charged with no order recorded and 12 were charged twice; all 49 were found by reconciling provider charges against our payments table and fixed by 2026-09-15.

Timeline (UTC).

Root cause. POST /checkout reserved inventory, charged the card and recorded the payment inside one transaction. A connection inside a transaction is unavailable to anyone else until it ends, and the inventory UPDATE holds a row lock until then. At 400 ms this was invisible. At 25 s, 40 concurrent checkouts held every connection, so every request on the service waited. Trigger: the provider's slowdown — outside our control, and certain to happen again.

Contributing factors.

• The payment timeout (30 s) equalled the load balancer's, so users got 504s while our code still waited, and many pressed "Pay" again.
• No idempotency key on charges, so a retry made a second charge (SD QA Q26).
• A timed-out charge rolled back our transaction, but the provider sometimes completed it anyway — the 37 charges without orders. The 14:17 restarts added to these.
• Checkout and read endpoints shared one pool, so slow checkouts starved unrelated pages.
• No idle_in_transaction_session_timeout was set.

Detection. The symptom alert fired 6 minutes after impact began, as designed. The saturation signal — waitingCount above zero — appeared 7 minutes earlier, unwatched (Chapter 10).

Response. Went well: deploy markers ruled out a rollback in two minutes; pg_stat_activity was saved before the restarts; updates went out every 15 minutes. Went badly: eight minutes on slow queries despite 15% database CPU (Chapter 9: check the pool first); two unannounced restarts worsened data impact; severity was raised late. Lucky: the provider recovered on its own; the database had spare connections; the code's author was online.

Action items.

Lessons. Never hold a transaction open across a network call. A timed-out call may still have succeeded, so external writes need idempotency keys and reconciliation. Restarts go through the IC like any other action.

Part 5: Telling the Story in an Interview

Debugging comes up in most interview loops in four forms: "Tell me about the hardest bug you've fixed" (your method on a real problem), "Tell me about a production incident" (how you behave under pressure, in a team), "How would you debug X?" (the method on a problem you haven't seen), and the live debugging round (the method happening, out loud).

The structure

Adapt STAR (Situation, Task, Action, Result) for debugging:

1. Situation — the system, the stakes, the scale, in two sentences.
2. Symptom — what was observed, precisely, with a number.
3. Investigation — your hypotheses, how you narrowed them, the instruments you used. This is what interviewers care about most; spend half your time here.
4. Root cause — the mechanism, physically.
5. Fix and prevention — the immediate fix, the proper fix, and what stops the class of bug.
6. Impact and reflection — numbers before and after, and what you would do differently.

What interviewers listen for, and common mistakes

They listen for method over luck (several hypotheses, cheap checks first, each result ruling something out); ownership ("I" for what you did, "we" for the team); communication; prevention; numbers ("p99 went from 9 s to 350 ms", not "much faster"); and honesty about mistakes ("I checked slow queries before the pool; now I check the pool first" is a strength).

The common mistakes are the opposites: "I just added a console.log and found it" (say why you put it there and what it ruled out); blaming others (it shows how you will talk about them); no numbers; drowning in jargon (name the mechanism plainly); a lucky-guess story (choose a different bug); and no prevention — "and then it worked" misses the senior half of the answer.

Honesty is not optional

Never present someone else's incident, or an invented one, as your own experience — not a colleague's, not one from a blog, not one from this track. Interviewers ask follow-ups ("what did the pool metrics show?", "who decided to roll back?"), and an invented story falls apart by the third. More importantly, it is a lie, and being caught ends the process. "I haven't been through a production incident yet, but here is a real bug I investigated, and here is how I would approach one" is a good answer.

If you have no production incidents yet, build real experience:

• Bugs from your own projects and side projects, described honestly at their real scale.
• Open-source issues you investigated — real debugging, with a public record.
• Bugs you deliberately reproduce in a lab. Hold a transaction open across a slow fake HTTP call, load-test it, and watch waitingCount and pg_stat_activity (Chapter 9). Or put a proxy in front of a Node server whose keep-alive timeout is shorter than the proxy's idle timeout and watch the 502s appear (Chapter 3). Then say exactly that: "I reproduced this in a lab to understand it."
• For "how would you debug X?", you need no story. Walk through the method from Chapter 1 using the casebook patterns.

Two example outlines

These are structures to fill with your own real experience, not scripts. Replace every bracket with what actually happened. If you have not done it, do it first or tell a different story.

Outline A — a bug you reproduced and fixed in your own project.

• Situation: [the project, its stack, its real usage — "a handful of users" is fine].
• Symptom: [what you saw, with a number — say, timeouts above N concurrent users in a load test].
• Investigation: [the first observation, such as low database CPU while latency rose]; [two or three hypotheses and why you ranked them]; [the instrument for each — pool counts, pg_stat_activity, a trace — and what each ruled out].
• Root cause: [the mechanism, in two sentences].
• Fix and verification: [the change]; [the same load test, numbers before and after].
• Prevention: [the test, timeout, alert or lint rule you added].
• Reflection: [what you would check first next time].

Outline B — "How would you debug intermittent 502s?" (from Chapter 13, Case 2, with Chapters 3 and 4)

• Clarify: what fraction, since when, which endpoints, seen by whom?
• Who sent the 502 (Chapter 4): the proxy, which got no valid response from upstream. Fast (refused or reset connection) or after a delay?
• Correlate with time and change (Chapter 10): clustered around deploys (shutdown not draining, Chapter 8), around restarts or OOM kills (exit code 137, Chapter 3), or spread evenly?
• Hypotheses, cheapest first: a keep-alive mismatch — the app closes idle connections before the proxy does (Node's default keepAliveTimeout is 5 s; many load balancers keep idle connections longer), so compare the two settings; crashes — restart counts and exit codes at the same timestamps; deploy-time drops — do they occur only during rollouts?
• Confirm, then fix and prevent: reproduce in staging or find the pattern in the proxy's logs; then set the app's keep-alive timeout above the proxy's idle timeout (check headersTimeout for your Node version), add graceful shutdown and an alert on the 502 rate.

Behaving in a live debugging round

• Think aloud. The interviewer can only grade what they hear; five silent minutes looks like being stuck.
• Restate, then hypothesise. "Expected 3 items, got 0. Either the filter is wrong, the data isn't seeded, or the query runs before the insert finishes."
• Ask for evidence. "Are there logs? Can I see the full error? Has this ever passed?" Interviewers often withhold information to see whether you ask.
• Narrate what each result rules out. "The data is there, so it's not seeding. That leaves the filter or the timing."
• If stuck, say what you know. "I've ruled out A and B; next I'd try C, because…"

The Causal Chains

The Rule You Keep Forever

In an incident, stop the harm, tell people, keep the evidence, then find the cause. Afterwards, fix the system rather than the person, with specific, owned action items. And when you tell the story, tell your own real one: what you saw, what you suspected, what each check ruled out, and what you changed so it cannot happen again.

Vocabulary Map

Practice Problems

Problem 1 — Declare or not? At 16:40 UTC you notice password-reset emails have stopped arriving for some users. Support has two tickets; nothing else is wrong. Do you declare? What severity, and what would change your answer?

Problem 2 — Write the update. A cache misconfiguration showed some users another user's order history. You turned the cache off and the leak stopped. You do not know yet how many users were affected. Write the internal status update using the four-part template.

Problem 3 — The rollback trap. A release added a status column, backfilled it and dropped the old state column. Errors spike ten minutes later and a teammate says "roll back." What do you check first, and what do you do if rollback is unsafe?

Problem 4 — Fix the postmortem. A draft says: "Root cause: an engineer ran a migration against production instead of staging. Action items: be more careful; improve monitoring." Rewrite the root cause and action items to this chapter's standard.

Problem 5 — Your story. Pick a real bug you investigated — at work, in a side project, in open source or in a lab. Fill in Outline A, then answer out loud the three follow-up questions an interviewer would most likely ask.

Where to Go From Here

That is the end of the track. Practice turns it into skill:

• Replay the investigations in Chapters 11–15 on the shortest path, then open every dead end and say why it was one.
• Do the practice problems out loud — speaking exposes the steps you skip in your head.
• Debug real things in your own projects. Write expected versus actual, list three hypotheses, run the cheapest check first. Reproduce a casebook failure on purpose — pool exhaustion, a keep-alive 502 — and investigate it with the real instruments.
• Time-box. Give yourself, say, 30 minutes before asking for help or changing approach, then write down what you have ruled out — half of a good question to a colleague.
• Keep a bug journal. For every non-trivial bug: symptom, hypotheses, what ruled each out, root cause, fix, time taken. Within months you will know your own blind spots — and you will have honestly earned the stories you tell in interviews.