Chapter 6 — Graceful Degradation

Deciding, before the outage, what users still get when a part of the system fails

Prerequisites: Chapter 3 — Isolation and Blast Radius (circuit breakers and bulkheads keep a failure contained; this chapter decides what the user gets instead) and Chapter 4 — Overload Protection (load shedding). Caching in depth: Debugging Ch 12 — Casebook: Caching, Q19 — Cache stampede, Q18 — Cache strategies. LLM fallback chains: AI Ch 9 — From Demo to Production. Circuit breakers: Q30.

How to read this chapter

When the interviewer says "your cache is down" or "recommendations take 10 seconds", the answer they want has two halves. The first half is containment: timeouts, breakers and bulkheads stop the failure spreading (Chapter 1, Chapter 3). The second half is what this chapter covers: what the user gets instead. A design that contains a failure but shows the user a blank error page has only done half the job.

Graceful degradation means the system keeps doing its most important work, with less, when a part fails. It is a design decision you make in advance, per dependency, not something you improvise at 3 a.m. Parts 1 to 6 go through the tools in the order you would use them: decide what is core, serve stale data, fall back, switch features off, go read-only, brown out. Part 7 puts them on one ladder.

All the code in this chapter runs on Node 22. The Postgres read-only results come from a real run on Postgres 18.

Part 1: Decide what is core before anything breaks

The availability arithmetic

If a request needs five dependencies and each is up 99.9% of the time, the request is up 0.999^5 = 99.5% of the time. That is about 3.6 hours down a month instead of 44 minutes. Every hard dependency multiplies in. Making a dependency optional takes it out of the multiplication: if the page can render without it, its outages are not the page's outages.

So the first move is to make the list short. For every call on the request path, write down what happens if it fails. Here is that table for a product page, and for an AI story generator like the reader's own app:

Two rules fall out of the table:

1. An optional dependency must never be able to fail a core request. That means a timeout shorter than the page's budget, a fallback value, and no unbounded retries. If it shares a connection pool with core calls, it can still take the page down by exhausting the pool, which is why it also needs its own pool (Chapter 3).
2. The table is the design. It belongs in the design doc and the interview whiteboard. In an incident, nobody should be deciding whether reviews matter.

The code: optional calls with a budget

// An optional call gets a small budget and a fallback. It can never fail the page.
async function optional(fn, ms, fallback, name) {
  try {
    return await fn(AbortSignal.timeout(ms));
  } catch (err) {
    metrics.increment('degraded', { dependency: name, reason: err.name }); // alert on the rate
    return fallback;
  }
}

async function productPage(id) {
  const product = await getProduct(id); // core: if this throws, the page fails
  const [recs, reviews] = await Promise.all([
    optional((signal) => getRecs(id, signal), 150, BESTSELLERS, 'recommendations'),
    optional((signal) => getReviews(id, signal), 150, null, 'reviews'),
  ]);
  return { product, recs, reviews };
}

I ran this (with fake dependencies, and console.warn in place of the metric) with a recommendations call that takes 10 s and a reviews call that throws a 500. The page returned in 172 ms: the product, the fallback list, and reviews: null. Without optional, the reviews error would reject the whole Promise.all at once, and on a day when reviews were healthy the page would wait the full 10 s for recommendations.

The signal matters. AbortSignal.timeout doesn't just stop waiting; when getRecs passes the signal to fetch, the request is cancelled and the socket is freed. A timeout that only stops waiting leaves the slow call running and holding a connection (JS Ch 6).

Part 2: Serving stale data on purpose

Most data shown to users can be minutes old during an outage without harm. A product description from 10 minutes ago is fine. An error page is not. Serving stale is choosing availability over freshness, per kind of data, in advance.

At the HTTP layer

RFC 5861 adds two Cache-Control directives:

Cache-Control: max-age=60, stale-while-revalidate=300, stale-if-error=86400

• max-age=60: fresh for 60 s.
• stale-while-revalidate=300: for 5 minutes after that, a cache may serve the old copy while it refreshes in the background.
• stale-if-error=86400: for a day, if refreshing fails (5xx, timeout, connection error), the cache may keep serving the old copy.

Support varies. Browsers mostly ignore stale-if-error, and CDNs differ in which of these they honour, so check your CDN's documentation and test it. On a reverse proxy you control, nginx does the same with proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504; plus proxy_cache_background_update on;. That alone turns many origin outages into "the site is a few minutes old" for public pages.

In the application: last-known-good

For data your code loads (from Postgres, an internal service, a config service), keep two ages: how long a value is fresh, and how long it is still worth serving if the source fails.

const entries = new Map(); // key -> { value, at }

export async function lastKnownGood(key, load, { freshMs, maxStaleMs }) {
  const e = entries.get(key);
  const age = e ? Date.now() - e.at : Infinity;
  if (age < freshMs) return { value: e.value, stale: false };
  try {
    const value = await load();
    entries.set(key, { value, at: Date.now() });
    return { value, stale: false };
  } catch (err) {
    if (age < maxStaleMs) return { value: e.value, stale: true, ageMs: age };
    throw err; // too old to be useful, or never loaded: fail honestly
  }
}

Run with freshMs: 50, maxStaleMs: 300 and a source that fails after the first load, it returned the fresh value, then { stale: true, ageMs: 102 }, then, once the copy was older than 300 ms, threw db down. In production the numbers would be something like 60 s fresh and 24 h max stale.

In Redis the same idea means storing the value with its own "fresh until" timestamp and setting the Redis TTL to the maximum stale age, not the fresh age. If the key expires at the fresh age, there is nothing left to serve when the database is down.

Three things go with serving stale:

• Tell the user when it matters. "Prices as of 10:02" or a small "Showing saved results" banner. Return the age in the API response (stale: true, ageSeconds) so the client can decide.
• Decide what must never be stale. Display can be stale. Decisions must check the source of truth at the moment they commit: the balance when spending credits, stock when charging the card, permissions when a user's access was just revoked. A cached "this user is an admin" that outlives a revocation is a security bug.
• Use last-known-good for configuration too. Feature flags, routing tables, rate-limit settings and model configs should load from the last good copy (in memory, and on disk for restarts) when the config service is down. And if a new config fails validation, keep the old one. A bad config push is one of the most common causes of large outages, and rejecting it is the cheapest defence.

When the refresh itself is the problem

Serving stale also stops a cache stampede: when a hot key expires, thousands of requests miss at once and all hit the database (Debugging Ch 12, Case 4). In the sim's second tab, answer the lock question first. Then, in free play, set Fix to SWR (stale-while-revalidate) and press Play. Requests get the old value instantly while one background query rebuilds it. Compare the runs table against None and Lock.

Part 3: Fallbacks, and how they go wrong

The fallback ladder

When a dependency fails and there is no stale copy, pick the highest rung that works:

For LLM calls, rungs 1 to 4 are the provider fallback chain described in AI Ch 9: the same request to a second provider, then a smaller model, with a breaker per provider so a degraded one is skipped quickly. A single interface over providers (GenAI JS Ch 1) is what makes switching cheap, and a durable job (GenAI JS Ch 12) is rung 4: if every provider is down, the story is generated later instead of lost.

Four ways fallbacks backfire

1. The fallback is heavier than the primary. "If Redis is down, read from Postgres" sends 50,000 reads a second to a database sized for the 2% that miss. "If provider A is down, use provider B" sends all your traffic to an account with a lower rate limit and a different price. A fallback needs its own limit: a concurrency cap on the database path, a budget on provider B.
2. The fallback has never run. Code that runs once a year is broken when you need it: the static file was deleted, the second provider's key expired. Amazon's Builders' Library has an essay, "Avoiding fallback in distributed systems", arguing that a rarely exercised fallback can make an outage worse. The fixes: exercise it continuously (send 1% of traffic through the fallback all the time), or prefer fallbacks that are trivially safe, like a static value.
3. The fallback hides the failure. If recommendations silently fall back for a week, nobody notices the service is broken. Count every fallback (the metrics.increment in Part 1) and alert on the rate.
4. The fallback picks the wrong default. A default that means "allow" on a security check is a hole. Choose fail open or fail closed per check:

Part 4: Kill switches

A kill switch turns one feature off at runtime, without a deploy. In an incident it is the fastest containment you have: a deploy takes 10 to 30 minutes, a switch takes seconds. Every feature that adds a new dependency or a new expensive path should ship with one.

A switch is only useful if it works while things are broken. That sets the requirements:

• Read from memory on each request. No network call per check. A poller (every 5–10 s) or a push stream updates the in-memory copy.
• Keep the last good copy on disk. If the flag service is down when a pod restarts, the pod starts with the last known values, not with defaults.
• Don't store switches only in the system they protect. A switch that lives in Postgres can't be read when Postgres is the problem.
• Safe defaults for a switch that was never set.
• Known names in the runbook. "Turn off recommendations" is a runbook step. "Find the flag for the recs thing" is not.
• Permissions and an audit log. Who flipped what, when.
• Tested. Flip each switch in staging regularly. A switch nobody has flipped is a fallback that has never run.

import { readFileSync, writeFileSync } from 'node:fs';

const DEFAULTS = { readOnly: false, recommendations: true };
const LKG_FILE = './switches.lkg.json';

let switches = (() => {
  try { return { ...DEFAULTS, ...JSON.parse(readFileSync(LKG_FILE, 'utf8')) }; }
  catch { return { ...DEFAULTS }; }
})();

export const isOn = (name) => switches[name];

export async function refreshSwitches(fetchSwitches) {
  try {
    const next = { ...DEFAULTS, ...(await fetchSwitches()) };
    switches = next;
    writeFileSync(LKG_FILE, JSON.stringify(next)); // last-known-good for restarts
  } catch {
    // flag service down: keep the values we have
  }
}

setInterval(() => refreshSwitches(fetchFromFlagService), 5_000).unref();

I ran it: after a refresh set readOnly: true, recommendations: false, a second refresh against a failing flag service left both values unchanged, and the file held the same values for the next restart.

A circuit breaker is a kill switch that flips itself. You want both: the breaker reacts in seconds to errors it can see, and the manual switch covers what it can't see, such as a feature that works but costs too much or writes bad data.

Part 5: Read-only mode

Read-only mode keeps every page that only reads working and rejects writes cleanly, with a clear message instead of timeouts and half-finished requests. It is the right answer more often than people expect:

• during a planned switchover or failback, while writes drain (Chapter 5);
• when the primary is lost and promoting a replica would lose data. Serving reads from replicas while humans decide is better than a full outage;
• during a risky migration;
• when the database disk is nearly full;
• when a bug is writing bad data. Stopping writes stops the damage spreading while you investigate (Chapter 7).

Building it: five layers

1. One switch. readOnly, from Part 4.

2. The API rejects writes with a clear error.

const SAFE = new Set(['GET', 'HEAD', 'OPTIONS']);

function readOnlyGuard(req, res, next) {
  if (isOn('readOnly') && !SAFE.has(req.method)) {
    res.set('Retry-After', '60');
    return res.status(503).json({
      error: 'read_only',
      message: 'Saving is paused for maintenance. Nothing you already saved is affected.',
    });
  }
  next();
}

I ran the plain node:http version of this: with the switch on, GET returned 200 and POST returned 503 with Retry-After: 60 and the read_only code. 503 with Retry-After tells well-behaved clients and providers to come back later. A machine-readable code lets the UI show the right message instead of "Something went wrong".

3. The UI says so. A banner, save buttons disabled, and drafts kept in local storage so nobody loses what they typed.

4. Background workers stop writing. Pause consumers and cron jobs that write. Messages wait in the queue instead of failing and being retried into a dead-letter queue.

5. A database guard. Postgres can refuse writes for every new transaction:

ALTER SYSTEM SET default_transaction_read_only = on;
SELECT pg_reload_conf();

I ran this on Postgres 18 with two sessions already connected, one of them in the middle of a transaction:

So the reload reaches existing pooled connections, in-flight transactions finish, and errors carry SQLSTATE 25006, which the app can map to its own read_only response. Two warnings from the run:

• It is a guard against mistakes, not a security boundary. Any session can turn it off for itself (last row). For real enforcement, revoke write privileges from the app's role, or send traffic to a replica: a hot standby rejects writes with the same 25006 error and cannot be talked out of it.
• A Postgres job queue stops completely. Claiming a job uses FOR UPDATE SKIP LOCKED and enqueueing uses nextval(). Both fail. If your queue lives in the same database, read-only mode also pauses all background work, so plan for the backlog.

The hidden writes

The hard part of read-only mode is the reads that secretly write:

• last_seen_at updated on every request;
• view counters and "recently viewed" lists;
• session or refresh-token rotation on each request, and login itself (last_login_at);
• "mark as read" when a notification list is fetched;
• audit logs written on reads.

Each one turns a read into a failed request in read-only mode. Make them best-effort (skip the write when readOnly is on), move them to Redis or a queue, or keep sessions in signed cookies. To find them, run your read-only test suite against a database user with default_transaction_read_only = on. Every failure is a hidden write.

And some writes can't wait. A payment provider's webhook must not be acknowledged and then dropped. Return 503 so the provider retries later; most providers retry webhooks for hours to days, but check yours. Don't return 200 for work you didn't save.

What it looks like during a 60-second failover

Part 6: Brownouts

Load shedding (Chapter 4) drops whole requests under overload. A brownout serves every request but does less work for each one. When it's possible, it's the better first step: users get a thinner page instead of an error.

The mechanism is about cost per request. If the optional parts of a home page (recommendations, "trending", the AI summary) use 40% of its CPU, turning them off lets the same servers handle 1 / (1 − 0.4) ≈ 1.67× the traffic. That can carry you through the 10 minutes autoscaling needs.

Order what you turn off by cost saved divided by value lost:

A brownout can be triggered by a person (the switches from Part 4) or by a controller: if p99 is above target or CPU above 75% for a minute, step down one level, and step back up only after 5 minutes of health. The wait before stepping up (hysteresis) stops the system from flapping between levels every few seconds.

There is a second, unrelated use of the word: a planned brownout for deprecations. Some API providers switch a deprecated endpoint off for an hour on announced dates, so callers that still depend on it find out while it can still be switched back on.

Part 7: The degradation ladder

Put the tools on one ladder, with a trigger and a switch for each level. This is what you draw in the interview when asked "what happens as things get worse?"

Each level should be a named switch in the runbook, and each should have been turned on at least once in a game day (Chapter 8).

Practise: where did the degradation fail?

When the Interviewer Breaks It

Break 1: "Your Redis cache is gone. What do users see?"

Break 2: "Recommendations now take 10 seconds. Does the home page load?"

Break 3: "The primary database is failing over. You have 60 seconds with no writes. What does the product do?"

Break 4: "Your main LLM provider is down."

Break 5: "Traffic just tripled. Autoscaling takes 10 minutes."

The Causal Chains

Checklist: add this to any design

• A dependency table: each call on the request path marked core, optional or deferrable, with what the user gets when it fails.
• Every optional call: a timeout under the page's budget, an abort signal, a fallback value, its own pool.
• A metric and an alert on fallback rate per dependency.
• Cache entries kept until the maximum stale age, with a separate fresh-until time. stale-while-revalidate and stale-if-error on public pages.
• A list of data that must never be served stale for decisions: balances, stock at purchase, permissions.
• Last-known-good for configuration and flags, in memory and on disk, and validation that rejects a bad new config.
• Fail open or closed chosen per check (auth closed, rate limiter open, spend cap closed).
• A kill switch for every feature with a new dependency or an expensive path, read from memory, named in the runbook, tested.
• Read-only mode: API 503 + Retry-After + read_only code, UI banner and local drafts, workers paused, database guard, hidden writes removed.
• A brownout order, with measured CPU per feature, and a controller with hysteresis.
• The degradation ladder (levels 0–4) with triggers, practised in a game day.

What's next

Chapter 7 — Data Safety: when the failure is not an outage but bad or lost data. Backups, point-in-time recovery, and restores you have actually tested.