L6 · Breaks in Production
The code passes every test. Then it meets real traffic, real failures, real deploys and months of uptime.
The story
On the morning of August 1, 2012, Knight Capital, one of the largest US stock-market traders, deployed new trading code to its eight servers. The deployment reached seven of them. The eighth still ran old code, including a long-retired feature called "Power Peg".
The new release reused a configuration flag that, years earlier, had switched Power Peg on. When the market opened, the seven updated servers used the flag for the new feature. The eighth server read the same flag and started Power Peg, which sent orders in a loop.
In about 45 minutes the firm had lost roughly $440 million, and it never recovered as an independent company.
None of this shows up in a unit test. The code was "correct". The bugs were in how it was deployed, what it assumed about its environment, and what happened to old code and old flags over time. That's L6: the bugs you see only when you picture the code in production, at scale, mid-deploy, months from now.
Part 1: The production lens
For every piece of code, run five pictures through your head:
| Picture | Questions | Typical bugs |
|---|---|---|
| Scale | What if this table has 50 million rows? This list 100,000 items? | No LIMIT, whole tables in memory, O(n²) loops |
| Failure | What if this call takes 30 seconds? Returns garbage? | No timeouts, unsafe defaults, brittle parsing |
| Deploys | What if old and new code run at once? What if the migration runs first, or second? | Renamed columns, locking migrations, reused flags |
| Time | After a month? On a server in UTC? Across a restart? | Unbounded growth, lost in-memory work, time-zone assumptions |
| 3 a.m. | When it fails, what do the logs and metrics say? | Context-free logs, swallowed errors, no correlation IDs |
Part 2: Fragility
Drill 1: A dependency client
Drill 2: Process lifecycle
Part 3: Deploys — old and new code at the same time
During a rolling deploy, old and new versions run side by side for minutes. Migrations run before, during or after. Queues hold messages written by the old version. Every change must work with both.
The standard technique for schema changes is expand and contract:
Each step is a separate deploy, and each one works with the version before it. No step renames or drops anything that running code still uses.
Drill 3: A release
Other deploy hazards to look for:
- Reused flags or enum values that mean something different to old code (the Knight Capital bug).
- Message formats changed while old messages are still queued, or old consumers still running.
- API fields renamed or removed while old mobile app versions still call them for months.
- Development settings (
.envloading, debug flags, permissive CORS) reaching production.
Part 4: Performance as an outage
A slow endpoint isn't just slow. In Node, CPU work on the main thread stops every other request on that instance. Quadratic code that takes 10 ms in testing can take minutes on real data.
| Pattern | At 1,000 items | At 100,000 items |
|---|---|---|
| O(n) loop | ~instant | ~instant |
O(n²): includes/indexOf inside a loop over the same data | ~1 million steps | ~10 billion steps: seconds to minutes |
| A sync crypto or file call in a handler | Every request on the instance waits for it |
Drill 4: Hot paths
Part 5: Assumptions about the world
Code written against the happy example assumes things about its inputs and environment that production will break: payload shapes, list lengths, time zones, ordering.
Drill 5: Assumptions
Part 6: When it breaks at 3 a.m.
Operability bugs don't cause the incident. They make it last longer: nobody can tell what failed, for whom, or why.
Drill 6: Can anyone debug this?
Drill 7: Months of uptime
Cheat sheet
| Picture | Look for | Fix |
|---|---|---|
| Scale | SELECT * with no LIMIT, filtering in JS, nested scans | SQL filters and limits, pagination, Set/Map lookups |
| Failure | Network calls without timeouts; defaults on error | Timeouts, safe defaults, fallbacks |
| Failure | Third-party payloads used without validation | Schemas; acknowledge and log what you can't handle |
| Deploys | Renames/drops in migrations; code needing new columns | Expand and contract; migrations before code |
| Deploys | Plain CREATE INDEX, SET NOT NULL on big tables | CONCURRENTLY, NOT VALID + VALIDATE |
| Deploys | Reused flags, changed message formats | New names; versioned messages |
| Lifecycle | No SIGTERM handling; one-shot DB connect; trivial health checks | Graceful shutdown, startup retries, readiness checks |
| Time | Map/Set/array that only grows; in-memory queues | Bounds and TTLs; durable queues |
| Time | getHours(), new Date() for business rules | Explicit business time zone |
| Event loop | *Sync( in handlers, quadratic loops, huge JSON.stringify | Async APIs, linear algorithms, small logs |
| 3 a.m. | console.log('failed'), swallowed errors, no request IDs | Structured logs with IDs and causes, metrics, propagated IDs |
Say it in the interview
"This migration renames
totalwhile six instances are rolling over, so for several minutes the old instances query a column that doesn't exist and every order fails. I'd split it into expand and contract: addtotal_paise, write both, backfill, switch reads, and droptotala release later. Same review for the index: it needsCONCURRENTLY, or writes to orders stop for the whole build."
Go deeper
- SD 6 · Break My Design: the same failures at the architecture level, as drills.
- SD 7 · Fault Tolerance and Idempotency: timeouts, retries, circuit breakers.
- Debugging · Production Observability: what good logs and metrics look like.
- FE 2 · The Main Thread and Scheduling: why blocking work stops everything (the same event loop).
- Next: Tests, Traps and Smells.