Chapter 14 — Casebook IV: "Nobody Changed Anything"

Time Bombs, Calendars, Drifting Dependencies, Flaky Tests and git bisect

Part 3: The Casebook. Prerequisites: Parts 1 and 2 (Chapters 1–10), and Chapter 11 for the case format.

The Changes Nobody Made

Chapter 1 made the claim this chapter rests on: systems that were working do not break on their own. When someone in the incident channel says "nobody changed anything," they mean "nobody changed anything we were looking at." The deploy log is empty, the config repository is quiet, and the system that worked at 23:59 fails at 00:01.

The change is real, just invisible to the tools you check first. Here is where it hides:

One habit helps with all of them: when a failure starts at an exact time, convert it to UTC. 05:30 IST is 00:00 UTC — when certificates expire, quotas reset and nightly jobs begin.

Case 1 — "Calls to One Partner API Started Failing at Midnight"

Symptom. Since 00:00 UTC, no order gets a shipping label. The worker logs request to partner failed. Nobody has deployed since Thursday. The partner's dashboard loads fine in Chrome.

The other ways the bomb goes off

Piping s_client into openssl x509 -noout -dates (Chapter 6) reads only the leaf; with -showcerts, inspect every block. openssl x509 -checkend 1209600 exits non-zero if the certificate expires within 1,209,600 seconds (14 days) — a one-line alert for a cron job.

Prevention has two layers. Alert on expiry of every certificate you serve or depend on, at 30, 14 and 7 days, checked from outside like a real client. And monitor the renewal job itself: the expiry alert says the bomb is close; the job alert says the day the fuse was lit. Keep an owned inventory of everything else that expires — API keys, OAuth client secrets, SAML signing certificates.

What an interviewer is testing: that you read the error's code, know a working browser proves little about server-to-server TLS, and refuse to disable verification under pressure.

Case 2 — "Inserts Started Failing on a Tuesday Afternoon"

Symptom. At 15:12 on a Tuesday, every POST /orders returns 500. Reads work. Nothing was deployed. The error tracker shows:

error: nextval: reached maximum value of sequence "orders_id_seq" (2147483647)

First five minutes. Read it literally (Chapter 2) and look up the SQLSTATE (Chapter 3): 2200H, sequence generator limit exceeded. If the sequence is bigint but the column is integer, the same event appears as integer out of range (22003) when 2,147,483,648 is written into the column.

Root cause. orders.id was created years ago as serial: a 32-bit integer (int4), maximum 2,147,483,647 (2³¹ − 1). Each insert takes the next sequence value; the counter climbed by about a million a day and arrived on Tuesday at 15:12. The change was 2.1 billion nextval() calls.

The table had only 1.3 billion rows, so nobody worried. Sequences are not transactional: a value taken by a rolled-back insert is gone, and INSERT … ON CONFLICT DO NOTHING consumes one even when no row is inserted.

Time to exhaustion

The number that matters is days until the sequence runs out:

SELECT sequencename, data_type, last_value, max_value,
       round(100.0 * last_value / max_value, 1) AS pct_used
FROM pg_sequences WHERE last_value IS NOT NULL
ORDER BY pct_used DESC;

Days left = (max_value − last_value) ÷ values consumed per day. Measure consumption from the sequence (record last_value daily as a metric), not from new rows, which miss the gaps. An events_id_seq at 2,050,000,000 consuming 3 million a day has about 97 million left: roughly 32 days. Trap: a bigint sequence feeding an integer column reports the bigint maximum, so check the column's type too.

Migrating to bigint with care

ALTER TABLE orders ALTER COLUMN id TYPE bigint rewrites the whole table and its indexes under an ACCESS EXCLUSIVE lock — no reads or writes until it finishes, which on a billion rows is hours. Every referencing column (order_items.order_id) must change too. The careful path (PostgreSQL Ch 18): add a nullable bigint column kept in sync by a trigger; backfill in small batches; build its unique index with CREATE UNIQUE INDEX CONCURRENTLY; repeat for referencing columns; then, in one short transaction with a lock_timeout (Chapter 9), swap the columns, move the primary key to the new index, and run ALTER SEQUENCE orders_id_seq AS bigint.

One application surprise: node-postgres returns int8 values as strings by default (they can exceed JavaScript's safe integer range), so order.id === 42 silently becomes false. In an outage, a known emergency measure is restarting the sequence at the negative end of the integer range — another 2.1 billion values — safe only if no code assumes IDs are positive. A tourniquet, not a fix.

Other thresholds data crosses on an ordinary day

• The planner changes its mind. As a table grows, statistics shift and the planner may switch from an index scan to a sequential scan. Same query, new plan (Chapter 9's auto_explain; PostgreSQL Ch 3).
• A payload passes a size limit. A saved cart grows past express.json()'s default 100kb or nginx's default client_max_body_size of 1m: 413 Payload Too Large, for one customer only — the tell (Chapter 4).
• Deep pages get slow. OFFSET 200000 reads and discards 200,000 rows (PostgreSQL Ch 13).

What an interviewer is testing: that you know the int4 ceiling, can estimate time to exhaustion, and know why a naive ALTER COLUMN TYPE on a large table is an outage of its own.

Case 3 — "The Nightly Job Ran Twice"

Time is an input every program has and almost no test controls.

UTC removes the ambiguity; idempotency makes the inevitable double run harmless — jobs also run twice after timeouts, failovers and manual re-runs (Chapter 13 Case 6; SD QA: job scheduler, idempotent writes).

The rest of the calendar

• "Add 24 hours" is not "same time tomorrow." A local day across a DST change is 23 or 25 hours. With TZ=Europe/Berlin, adding 24 * 3600 * 1000 ms to 09:00 on 28 March 2026 gives 10:00 on 29 March; d.setDate(d.getDate() + 1) keeps 09:00. A token lifetime is a duration; "same time tomorrow" is a calendar step.
• Month arithmetic. 31 January plus one month: JavaScript's Date overflows "31 February" to 3 March (in 2026); many date libraries clamp to 28 February instead. Billing on the 31st needs an explicit rule.
• 29 February. "Same date next year" turns 29 February 2028 into 1 March 2029 (new Date(2029, 1, 29) rolls over). Leap-day bugs ship in an ordinary year and surface years later.
• The week-numbering year. ISO week 1 is the week containing the year's first Thursday, so 29–31 December 2025 belong to week 1 of 2026. Formatting libraries distinguish calendar year from week-based year, with different letters: in Unicode-style patterns (Java's formatters, ICU, date-fns) yyyy is the calendar year and YYYY the week-based year; in Moment.js YYYY is the calendar year and GGGG the ISO week year. The wrong token shows next year for a few days around 31 December. A test dated 30 December catches it; a test using today never will.
• Server zone vs user zone. "Today" on a UTC server is not "today" in Sydney for part of every day; daily limits and "due today" filters must say whose today (Chapter 11 Case 3).
• Expiry dates nobody wrote down. One-year API keys, CI access tokens, refresh tokens that lapse after inactivity. They fail like Case 1: an exact moment, an auth error, no deploy.
• Leap seconds have caused outages (2012) but are rare and due to be abandoned by 2035; clock skew between machines (Chapter 11 Case 5) is far more common.

The rule for time: schedule in UTC, store instants in UTC, keep calendar dates as dates. Convert to local time only at the edge — for display, or when the customer's calendar defines the rule ("09:00 in their zone"); then store the zone name (Europe/Berlin), not an offset (+01:00), which changes twice a year.

What an interviewer is testing: that you know local time is ambiguous, pair UTC scheduling with idempotency, and can name calendar edge cases unprompted.

Case 4 — "The Build Broke and Nobody Touched package.json"

Symptom. Monday's CI build of main fails with TypeError: Cannot read properties of undefined (reading 'parse') in a date helper. Friday's build of the same commit passed.

First five minutes. Same commit, different result: the difference is outside the code. Open both CI logs side by side and compare the resolved versions, the install command, whether a lockfile exists, and the base image digest each build pulled.

Root cause. package-lock.json was in .gitignore, so CI ran npm install from package.json alone. A direct dependency declared ^3.6.1 for one of its own dependencies. On Saturday that package published 3.7.0, a minor release that nonetheless removed a deprecated export. Nothing in your repository changed; the meaning of ^3.6.1 did.

Semantic versioning (major.minor.patch) promises minor and patch releases are compatible — a promise made by people. A lockfile records the exact version of every package in the tree, transitive ones included. npm install may update the lockfile when it and package.json disagree. npm ci installs exactly what the lockfile says, deletes node_modules first, never writes either file, and fails if the lockfile is missing or out of sync. That failure is the feature.

How to investigate drift. (1) Rebuild the old commit today — if Friday's commit now fails, the change is outside the repository. (2) Diff lockfiles between last good and first bad (git diff <good> <bad> -- package-lock.json), or the resolved versions in CI logs. (3) Find who pulls it in: npm ls <pkg> shows every path to it; npm explain <pkg> says why it is installed. (4) Pin and confirm with an npm overrides entry.

The same drift, a layer down. FROM node:20 means whatever 20.x and OS packages were published most recently (latest is worse); pin by digest, FROM node:20.17.0-bookworm-slim@sha256:<digest>, which names exactly one image forever. Unpinned apt-get install, npx without a version, and CI actions on moving tags drift the same way.

Fix and prevention. Commit lockfiles, use npm ci in CI, pin base images by digest — then let Dependabot or Renovate open pull requests for new versions, so every upgrade arrives deliberately, with its own CI run, and a break lands on a pull request instead of a Monday morning.

What an interviewer is testing: what a lockfile is for, npm install versus npm ci, and that "same commit, different result" points outside the repository.

Case 5 — "A Third-Party API Changed Under Us"

Symptom. For a week the CRM sync has been "a bit off": some contacts are missing, and the nightly job sometimes takes three times as long. No errors. Nothing changed on your side.

First five minutes. Work from the boundary outward: the vendor's status codes and latencies per day (Chapters 7 and 10), their changelog and status page, and a raw response today against one from last month.

Root cause. Two vendor changes, announced in a changelog nobody read. Retries absorbed the 429s, so the lower limit showed only as slow runs. The smaller default page size became missing data with no error, because the sync relied on the default instead of following the pagination cursor to the end.

The family. Deprecations — some APIs announce removals with a Sunset header (the removal date) or a Deprecation header; log them. Changed defaults — page sizes, sort orders, API versions (APIs versioned by date or header often apply the account's default if you send none); send every parameter you depend on. Rate limits — honour Retry-After, add jitter, budget retries (SD QA: rate limiting, retry systems). Their outages — status pages lag your own per-dependency error rate; a circuit breaker stops a struggling vendor consuming your workers.

Fix. Request the page size explicitly and paginate until the vendor says there is no next page; reduce concurrency and honour Retry-After; re-sync the missing contacts.

Prevention — make the vendor's changes visible:

1. Validate responses at the boundary against a schema (for example with zod), failing loudly — the Chapter 1 worked example broke on exactly this.
2. Contract tests against the vendor's sandbox, on a schedule, asserting the fields and pagination you depend on.
3. Chart each dependency's status codes and latency; a new band of 429s is visible the day it starts.
4. Alert on 4xx from dependencies. A vendor's 4xx to you means your requests are now wrong in its eyes — a changed contract, revoked key or new limit.
5. Assert completeness where the vendor reports a total count.

What an interviewer is testing: that you treat a vendor as change you do not control, and know the dangerous failures are silent — wrong data with a 200.

Case 6 — "It Crashes Every Friday"

Symptom. The API dies about once a week, late in the week. Restarting fixes it. The errors differ: ENOSPC: no space left on device, then EMFILE: too many open files, then the container killed with exit code 137.

First five minutes. Put crash times next to deploy and process start times (many Prometheus client libraries include process_start_time_seconds in their default process metrics). The team deploys every Monday; every crash came four to five days later — and the week a Wednesday hotfix restarted everything, the crash arrived the next Monday.

The tell: failure correlates with uptime, not with a change. "Fails every Friday" means "we deploy on Mondays, and it takes four days to fill up."

Root cause — three slow leaks:

• Disk (ENOSPC). Logs went to a file with no rotation (Docker's default json-file driver also keeps output without limit unless max-size is set). Check df -h and du -sh /var/log/* | sort -h. Trap: deleting a log a process still holds open frees nothing until it closes it — lsof +L1 lists such files. df -i shows inode exhaustion: free space, but no room for new files.
• File descriptors (EMFILE). Every socket and file uses a descriptor, up to the process limit (cat /proc/<pid>/limits); ls /proc/<pid>/fd | wc -l counts them. An HTTP agent created per request and never closed leaked about 100 sockets an hour (Chapter 3).
• Memory (exit 137). A cache keyed by request ID with no eviction grew about 80 MB a day until the OOM killer sent SIGKILL. Post-GC heap troughs rose steadily; two heap snapshots hours apart show what grows (Chapter 8; Chapter 13 Case 5).

Fix. Rotate and cap logs (or write to stdout for the platform to ship); reuse one keep-alive agent; bound the cache (SD QA: eviction policies).

Prevention. Alert on trends, not only thresholds: "disk full in 48 hours at this rate" fires days before "disk 95% full." Export open descriptors and post-GC heap as metrics. Scheduled restarts are a legitimate mitigation, but they hide the leak until it outgrows them.

What an interviewer is testing: that you plot failures against uptime, and know the error each kind of exhaustion produces.

Case 7 — "The Test Is Flaky"

A flaky test passes and fails on the same code. It is never random: it depends on an input it does not control — order, timing, the clock, the network, randomness, shared state. It is "nobody changed anything" in miniature.

Symptom. user-profile.test.ts fails on CI about one run in ten with expected 'Ada' but received 'Grace'. Alone, it always passes. Re-running the pipeline turns it green.

Techniques, cheapest first. Run it many times: for i in $(seq 1 100); do npx jest user-profile -t 'renames' || break; done (Playwright has --repeat-each). Alone vs in the suite: green alone, red in the suite means order dependence — bisect the suite until you find the one test that poisons it (Chapter 1). Randomise the order: recent Jest versions accept --randomize (with --seed to replay); Vitest has sequence.shuffle. Freeze the clock with jest.useFakeTimers() and jest.setSystemTime(…) (Vitest: vi.useFakeTimers(), vi.setSystemTime(…)) and pick the nasty dates on purpose — month ends, DST nights, 29 February, 30 December. Seed randomness. Intercept or record the network (MSW, nock).

The policy. Once red sometimes means nothing, people re-run real failures too. Quarantine the test — out of the blocking suite — with a named owner and a deadline to fix or delete it. Never let "retry three times" become permanent: sometimes the flakiness is a real race in production code that the test was right to catch.

What an interviewer is testing: that you can name the causes, reproduce one on purpose instead of re-running CI, and have a policy.

Case 8 — "Fine at v2.3. Broken Somewhere in 200 Commits."

Symptom. Invoice totals are off by one cent for orders with three or more discounted items. Release v2.3.0, six weeks ago, was correct; main is wrong. 200 commits lie between, and nobody remembers touching the rounding code.

Instead of reading 200 diffs, let Git bisect — binary search over history (Chapter 1). Each test halves the suspects: 200 commits need about log₂ 200 ≈ 8 tests.

git bisect start
git bisect bad              # current checkout (main) is bad
git bisect good v2.3.0      # this tag or SHA was good
# Git checks out a midpoint; build, test, then mark it:
git bisect good             # or: git bisect bad / git bisect skip
# ...until "<sha> is the first bad commit"
git bisect reset            # return to where you started

git bisect start <bad> <good> does the first three lines in one; git bisect log records your marks. For "slower" rather than "broken", git bisect start --term-old=fast --term-new=slow lets the words match the question.

Automating it: git bisect run

Give Git a script whose exit code is the verdict: 0 good; 1–127 except 125 bad; 125 skip (cannot test); anything above 127 aborts the bisect.

#!/usr/bin/env bash
# /tmp/check-rounding.sh — lives OUTSIDE the repo, so checkouts don't change it
npm ci --silent        || exit 125   # can't install this commit: skip it
npm run build --silent || exit 125   # can't build it: skip it
node /tmp/rounding-repro.mjs         # exits 0 if the total is right, 1 if it's off by a cent

git bisect start main v2.3.0
git bisect run /tmp/check-rounding.sh
git bisect reset

Three details matter. Keep the script and reproduction outside the working tree — checkouts replace its files, and old commits lack your new test. Make the reproduction specific to this bug, or an unrelated failure mid-history sends the search the wrong way. And map "could not test" to 125, not 1, or an unbuildable commit takes the blame.

Bisecting things that are not code. Any ordered history with a testable point works: lockfile changes (revert half of a 40-package upgrade; Case 4), configuration history, data (the first 5,000 of 10,000 rows), infrastructure (one canary on last week's image).

What an interviewer is testing: that you reach for binary search over a large ordered space, know it needs only a good point and a reliable test, and can automate it.

The Pattern Behind Every Case

Every case began with "nobody changed anything" and ended with a specific change. The method: assume something changed, go down the list of invisible changes, and make the suspect visible — a date, a number, a diff, a graph.

The Causal Chains

The Rule You Keep Forever

"Nobody changed anything" is never true. Time passed, data grew, a version resolved, a vendor shipped, a counter climbed. Assume there was a change, list the ones you cannot see, and make your suspect visible — as a date, a number or a diff — before you read a single line of unchanged code.

Vocabulary Map

Practice Problems

Problem 1 — Read the clock. An outage began at 05:30 IST on the first of the month. List four kinds of invisible change scheduled for exactly that moment, and the first check for each.

Problem 2 — Days left. pg_sequences shows invoices_id_seq as integer with last_value 1,912,000,000; a week ago it was 1,898,000,000. How many days until inserts fail? Describe the migration, and what might break afterwards in a Node API using node-postgres.

Problem 3 — The schedule. A "daily digest" job runs at 0 1 * * * in America/New_York; a usage-billing job charges "every 24 hours from sign-up" by adding 86,400 seconds. What does each do on the US DST nights, is it a bug, and what would you change?

Problem 4 — Same commit, new failure. An image built from an unchanged commit now fails at startup in a native image-processing module. What do you compare first, which commands do you run, and which three things do you pin afterwards?

Problem 5 — Bisect it. A page became 600 ms slower somewhere in the last 350 commits; the benchmark varies by ±80 ms on one commit. Write the git bisect commands and script, handling unbuildable commits and keeping the noise from misleading the search.

What's Coming Next

One more casebook first. Chapter 14B takes the method to the UI: React bugs that come from how rendering works — render loops, stale closures, wrong keys, out-of-order responses — and the React Native bugs that only happen on a phone or in a release build.

After that, Chapter 15, the last, is about everything around finding causes: running an incident while the cause is unknown, communicating while you investigate, choosing mitigation over diagnosis, writing a blameless postmortem that produces real prevention — and telling the story of a hard bug clearly in an interview.