Break My Design — Course Map
Any design works when nothing fails. This section is about what yours does when something does.
Read this first. It takes ten minutes. It assumes you can already draw a reasonable design for a common system; SD 2 Chapter 19 — The System Design Framework covers that.
The Move You Will See
At some point in a strong system design round, the interviewer stops asking you to add things and starts taking them away.
- "Your cache cluster just went down."
- "The primary database died."
- "That recommendations service now takes ten seconds to respond."
- "One customer is sending a hundred times their normal traffic."
- "You shipped a migration an hour ago and data is wrong."
They are not trying to trick you. A design that only works when every box is healthy tells them nothing, because in production some box is always unhealthy. What they are scoring is whether you can see the failure travel through your design, stop it from spreading, keep users served while it lasts, and change the design so it can't happen the same way again.
This section trains exactly that move. Chapters 1–8 cover the ten failure and reliability fundamentals. Drills 9–14 hand you a design and break it, five times each, the way an interviewer would.
The Answer Shape: Detect, Contain, Degrade, Recover, Prevent
When something breaks, answer in this order. It's the order an on-call engineer works in, so it sounds like experience.
| Step | The question it answers | What a strong answer names |
|---|---|---|
| Detect | How do we know, and how fast? | The metric or alert that fires: error rate, p99, queue depth, replication lag. Seconds, not "users complain". |
| Contain | How do we stop it spreading? | Timeouts, circuit breakers, bulkheads, load shedding: the thing that stops one sick box from taking the healthy ones with it. |
| Degrade | What do users get meanwhile? | Stale data, a fallback, read-only mode, a feature switched off. A worse experience on purpose, instead of an outage by accident. |
| Recover | How do we get back to normal without a second incident? | Failover, cache warm-up, draining a backlog at a controlled rate, rollback, restore. |
| Prevent | What changes in the design? | The concrete change, its cost, and why it's worth it. |
Then say the trade-off and a number. "We'd serve product pages up to five minutes stale during a cache outage, because a stale price for five minutes costs less than a checkout outage" is a senior answer. "We'd add redundancy" is not an answer.
The first thirty seconds
When the interviewer breaks something, don't jump to the fix. Say what users see, in order:
"In the first few seconds every request that hit the cache now goes to Postgres. We were at 50k reads a second with a 95% hit ratio, so the database goes from 2.5k to 50k reads a second, about 20 times its normal load. It will saturate its connection pool in well under a minute, and then even requests that don't need the cache start failing. So the first problem isn't the cache, it's protecting the database."
That answer traces the failure through the design with numbers, and finds the second thing that breaks. That second thing is usually what the interviewer was waiting for.
For every box you draw, ask three questions
- What happens if this box is down? Who notices, what fails with it?
- What happens if this box is slow? Slow is often worse than down: callers wait, hold resources, and pile up.
- What happens if this box is wrong? Stale data, duplicates, a bad deploy, a corrupted write.
If you ask these before the interviewer does, the round goes very differently.
The Ten Fundamentals, and Where Each Is Covered
The reader already covers much of this in depth, spread across many chapters. This table is the index. Each chapter of this section summarises what the deep chapters say and goes deep on what they don't.
| # | Topic | This section | Deep chapters elsewhere |
|---|---|---|---|
| 1 | Timeouts & deadlines: connection vs request timeouts, deadlines that travel across calls, what happens with no timeout | Ch 1 | Q33 — Timeouts, deadlines, retry budgets · Debugging 13 |
| 2 | Retries: backoff with jitter, retry budgets, retry storms | Ch 2 | Q29 — Retry systems · JS 7 — Controlling concurrency |
| 3 | Idempotency: keys for write APIs, at-least-once delivery, safe vs unsafe retries | Ch 2 | Q26 — Idempotent write APIs · Q43 — Delivery guarantees · SD 2 Ch 23 — Payments |
| 4 | Isolation: circuit breakers, bulkheads, blast radius per tenant, region and feature | Ch 3 | Q30 — Circuit breaker |
| 5 | Overload protection: rate limiting, backpressure, load shedding and priority | Ch 4 | SD 2 Ch 17 — Rate limiting · Q32 — Backpressure |
| 6 | Replication & failover: leader election and split brain, sync vs async replication, failover vs failback | Ch 5 | SD 2 Ch 4 — Why machines die · SD 2 Ch 5 — Leader-follower replication · SD 1 — Leader election & Raft |
| 7 | Graceful degradation: serving stale on purpose, fallbacks and feature switches, read-only mode | Ch 6 | Debugging 12 — Caching casebook · AI Ch 9 — From demo to production |
| 8 | Data safety: write-ahead logs, backups and point-in-time recovery, restores you've actually tested | Ch 7 | Q13 — Write-ahead log · SD 1 — Replication, WAL, lag |
| 9 | Observability: p50 vs p99, SLIs, SLOs and error budgets, alerts that map to a runbook step | Ch 8 | SD 2 Ch 18 — Observability · Debugging 10 — Production observability |
| 10 | Recovery & operations: rollbacks and feature flags, zero-downtime migrations, postmortems that change the system | Ch 8 | Debugging 15 — Incidents and postmortems · PostgreSQL First Principles |
Before this section, the reader was thin or silent on: connection vs request timeouts, safe vs unsafe retries, bulkheads, per-tenant and per-region blast radius, load shedding by priority, failback, read-only mode, tested restores, zero-downtime migrations and chaos testing. Those get the most space here.
The Drills
Each drill is one 45-minute round. You get the prompt and a baseline design, the kind a decent candidate draws in fifteen minutes and that is deliberately not over-built. Then the interviewer breaks it five times, each break harder than the last.
| Drill | The design | What gets broken |
|---|---|---|
| 9 — The Cache Dies | A read-heavy service with Redis in front of Postgres | The cache cluster, a hot key, the cold start, stale data on return |
| 10 — The Primary Dies | A checkout service on Postgres with async replicas | The primary, a lagging promotion, split brain, failback, a region |
| 11 — One Downstream Takes 10 Seconds | A feed API that fans out to five services | A slow dependency, then an erroring one, then retries, then the shared pool |
| 12 — The Queue Backs Up | A notification and webhook delivery system | A consumer outage, a poison message, a rate-limited provider, duplicates, the recovery flood |
| 13 — One Tenant Takes Everyone Down | A multi-tenant AI generation SaaS | A 100× tenant, an LLM provider outage, the spend cap, a lost region |
| 14 — The Bad Deploy | Any service, one hour after a deploy | A bad migration, corrupted data, an impossible rollback, an untested restore |
How to run a drill
- Read only the prompt and the baseline design. Set a 45-minute timer.
- At each break, stop. Answer the "Before you read on" questions out loud, in the Detect → Contain → Degrade → Recover → Prevent shape, with numbers.
- Use the timeline's predict gate and the sims. Commit to a prediction before you step through.
- Then open the answer and compare. Note what you missed, not what you got.
- Score yourself with the rubric at the end of the drill.
Drill 13 is built to look like the system you've actually run: a job queue, LLM provider failover, a credit ledger and a spend cap. Expect to be asked exactly those breaks about your own work. If you can answer that drill for Prajna, you can answer it for anything.
Suggested Order
If your interviews are a few weeks away:
| Week | Read | Drill |
|---|---|---|
| 1 | Chapters 1, 2, 3 | 11 — Slow downstream, then 9 — Cache dies |
| 2 | Chapters 4, 5, 6 | 10 — Primary dies, 12 — Queue backs up |
| 3 | Chapters 7, 8 | 13 — Noisy tenant, 14 — Bad deploy |
Then redo the drill you scored worst on, cold, a week later.
The Causal Chains
- If a design has no timeout on a downstream call
- If every layer retries three times
- If the cache is a correctness dependency rather than an optimisation
- If replication is asynchronous
- If nothing decides in advance what is optional
- If restores have never been tested
What's Next
Start with Chapter 1 — Timeouts and Deadlines. Every other failure in this section gets worse when a timeout is missing.