Drill 9 — The Cache Dies

What actually happens with the defaults. ioredis keeps an offline queue by default (enableOfflineQueue: true) and retries each command up to maxRetriesPerRequest: 20 times, with reconnect delays of min(attempt × 50, 2000) ms. Against a dead port, a single GET took 10.5 s to fail when I ran it on ioredis 5. So for 10 seconds nothing errors and nothing loads: each pod parks 2,000 × 10.5 ≈ 21,000 requests. Then they all fail over to the DB.

The DB arithmetic: 50,000 reads/s offered, 6,000/s possible, so 12% could be served. Each pod gets 6,000 / 25 = 240 q/s through its 10 connections, and 2,000 − 240 = 1,760 requests/s pile up in the pool's wait queue. pg's pool waits forever by default (connectionTimeoutMillis: 0). After one second of this, a new request waits 1,760 / 240 ≈ 7 s. Shoppers give up after a few seconds, so the DB spends its 6,000 q/s on requests nobody is waiting for. Useful throughput is near zero, not 12%.

Detect. Redis client error rate; cache hit ratio falling from 98% to 0; pg pool wait-queue length above zero; DB CPU at 100%; p99 above the 100 ms SLO. The alert that maps to a runbook step is "hit ratio below 80% for 1 minute": its first step is "check that the Redis breaker is open".

Contain. Make Redis fail fast: commandTimeout: 50, enableOfflineQueue: false, maxRetriesPerRequest: 1. With those settings the same GET failed in 4 ms. Wrap Redis in a circuit breaker (open after 50% errors over 5 s, probe every 5 s) so pods stop paying even the 50 ms. Bound the DB path: connectionTimeoutMillis: 100 on the pool, so a request waits at most 100 ms for a connection and then takes the fallback. This is a queue timeout: drop requests that have already waited too long (Chapter 4).

Degrade. Add an L1 cache: an in-process LRU in every pod. 100 k products × 2 KB ≈ 200 MB per pod, TTL 30 s. It covers 90% of reads (the popularity numbers in the prompt). With Redis gone, DB load is 50,000 × 10% = 5,000 q/s, 83% of capacity, so every request is served. Let L1 serve stale on error for up to 10 minutes: if the DB refresh fails, return the old entry. Shoppers see prices up to 30 s old, which the prompt allows for display.

Recover. Don't flip Redis back on at once; Break 3 is about why.

Prevent. One replica per Redis shard in another availability zone, so a node loss is a failover of a few seconds, not an outage. The client settings and L1 above. A game day in staging that kills Redis at peak load and checks that p99 stays under 100 ms (Chapter 8).

Trade-off to say out loud. L1 means 25 independent copies, each up to 30 s stale, and a write's DEL doesn't reach them. That is acceptable for display. It is not acceptable for the price you charge, so checkout reads price from the primary (more in Break 4).

Not by default. Redis Cluster has cluster-require-full-coverage yes as its default: when any hash slot has no node serving it, every node stops answering queries, so losing one shard looks exactly like losing all three. Two fixes: give each shard a replica, so a primary loss triggers a failover instead of uncovered slots; and set cluster-require-full-coverage no for a cache, so the surviving shards keep serving their two-thirds.

With that setting, a third of keys miss. Without L1: 50,000 × 1/3 ≈ 16,700 q/s, 2.8× DB capacity, so you are back in the main break. With L1: only L1 misses reach Redis, so 50,000 × 10% × 1/3 ≈ 1,700 q/s extra. Even on top of the normal 1,000 misses that is at most 2,700 q/s, 45% of capacity. Partial failures are where the L1 earns its memory.

What the sim shows. With no protection the key was missing for 0.45 s, not 0.25 s, and the DB received 950 queries for one value, peaking at 2,000 q/s, 10× capacity. 750 requests timed out. The rebuild took longer than 250 ms because it shared the database with up to 99 duplicates of itself: a stampede slows down the one query that would end it.

Detect. A spike in DB queries per second with the same query fingerprint (pg_stat_statements calls for one statement jump); the cache miss count for one key; p99 spikes on a regular beat, every TTL.

Contain. Single-flight: only one request rebuilds a key; the rest wait for its result. Across pods, take a lock in Redis with SET lock:product:8812 <token> NX PX 2000; inside a pod, share one in-flight promise per key so a pod sends at most one lock attempt. In the sim, Lock turned 950 queries into 1 query; 499 requests waited 112 ms on average (worst 225 ms) and 0 timed out.

Degrade. Stale-while-revalidate (SWR): store a soft expiry inside the value and keep the Redis key for longer (say soft 5 s, hard 60 s). After the soft expiry, every reader gets the old value at once and one background refresh rebuilds it. Nobody waits. The cost: stock can be one rebuild older than the TTL, so 5.25 s here. That is fine for "In stock" on a product page, because checkout re-checks stock on the primary anyway.

Recover. Nothing to recover once the herd is gone. The duplicates already running in Postgres finish on their own; the database doesn't know their callers left.

Prevent. Make the rebuild cheap. A 250 ms join over 40 warehouses is a read-time computation of something you can compute at write time: keep a stock_summary row that the stock service updates on every change, and the rebuild becomes a 2 ms primary-key read. Then the stampede is 2,000 × 0.002 = 4 queries, a non-event.

Trade-off. The lock adds latency for waiters (up to one rebuild) and needs a lock TTL in case the holder dies. SWR adds no latency but serves stale data. Early refresh (refresh with a small probability before expiry) needs no lock and serves fresh data, but a few duplicate refreshes can fire.

They wait until the lock's PX 2000 expires, then one of them takes it and rebuilds. That is up to 2 s of extra wait, which is why waiters need their own deadline: wait at most ~300 ms (just over one rebuild), then serve the stale value if there is one, or read the DB directly if there isn't. The lock value is a random token, and the holder deletes the lock only if it still holds that token (a small Lua script does the check and the DEL atomically). Otherwise a slow holder whose lock expired could delete the next holder's lock. Set the lock TTL from the rebuild's p99, not its average: 2 s against a 250 ms query is plenty.

Detect. The same hit-ratio alert as Break 1. The point is to not need it: recovery is a planned operation with its own runbook, not a switch you flip.

Contain. Keep the Break 1 protections on while you warm: the DB limit and the 100 ms pool wait, so even a bad cutover sheds load instead of collapsing.

Degrade. While warming, pods serve from L1 (refilling) plus the DB at its limit. If the L1s are also empty, as here, shed low-value reads first: listing tiles can show a placeholder; product detail pages, where people buy, get the DB capacity.

Recover. Warm the shared cache first, then ramp:

1. Free some DB capacity. Lower the pods' DB limit from 10 to 8 connections each. The pods keep about 4,800 q/s (their L1s refill from it) and about 1,200 q/s is left for warming.
2. Pre-warm Redis, not the L1s. A job reads the top 100 k product IDs from the last hour's access log and loads them into Redis at 1,200/s: 100,000 / 1,200 ≈ 85 s. One Redis fill serves all 25 pods. Filling 25 L1s straight from the DB would cost 25 × 100,000 = 2.5 M queries, about 7 minutes of the whole DB.
3. Ramp. Turn Redis reads on for 10% of pods with a feature flag, then 50%, then 100%, about 5 minutes in all, and stop if DB CPU passes 85%. The L1s now refill from Redis in 1 ms instead of from the DB.
4. Jitter the TTLs. Write warmed keys with TTL 1 h ± 20%. Otherwise 100 k keys written within the same 85 seconds all expire within the same 85 seconds an hour later, and you get Break 2 on every one of them.

Prevent. One replica per shard (so restarts are failovers and Redis rarely comes back empty), a warm-up job that is tested, and a runbook rule: never restart every pod at once during an incident, because it throws away 25 warm L1 caches at the moment you need them most. Roll restarts in batches.

Trade-off. A slow ramp means minutes of degraded service you could have skipped if the database had been bigger. Sizing the database to survive a full cold cache (50,000 q/s, 8× today) costs far more than a 5-minute ramp. Say which you'd pay for.

With identical TTLs, the keys expire together: in the sim the DB was asked for 2,080 q/s, 10× capacity, and 950 requests timed out. With ±40% jitter, the same 40 expiries spread over 6.3 s, the peak was 176 q/s and nothing timed out. Jitter doesn't remove any work. It stops the work arriving at once. Notice also that the top row, refilled in one burst, expires in one burst again: synchronised expiry repeats itself until something breaks the rhythm. For the real catalogue: 100 k keys expiring inside 85 s is more than 1,000 extra misses per second on top of normal traffic, at a moment the cache is otherwise healthy. Jitter of ± 20% on 1 h spreads them over 24 minutes: about 70 per second.

Why it happens. A snapshot stores each key with its absolute expiry time. On restart, Redis loads the keys whose expiry hasn't passed, with their old values. I checked this on Redis 8: a key snapshotted at price 1999 and then set to 1499 came back as 1999 with its TTL still counting down. So shard 2 lost two windows of invalidation: 15 minutes between the snapshot and the crash (the DELs landed, then the restore undid them), and 5 minutes of downtime (the write path did redis.del(key).catch(() => {}), so the DELs were dropped). That's 300 writes/s × 1,200 s = 360,000 updates in the window; about a third are on shard 2's keys. Each wrong value lives until its TTL: up to 1 h.

Detect. A cache that returns stale data looks healthy, so you detect events and drift, not errors: alert on any Redis restart or role change (uptime reset, failover), because after one the contents are suspect. Run a staleness sampler: every minute, read 200 random cached products, compare their version with the DB, and alert if more than 0.1% disagree. And count price mismatches at checkout (the cart's displayed price ≠ the primary's price).

Contain. You know the window exactly: from the snapshot time to now. Replay the invalidations: SELECT id FROM products WHERE updated_at > :snapshot_time - interval '1 minute', then DEL those keys in batches. Even if all 360,000 updates touched different products, at 20,000 DELs/s that takes 18 s. A blunter option is to bump the key namespace (catalog:v7: → catalog:v8:): every key is invalid at once, which is safe but puts you back in Break 3's cold start.

Degrade. Already covered if checkout is designed right: the cache is for display; checkout reads price and stock from the primary. The customer who added 1999 to the cart is shown "price changed to 2499" before paying. Nobody is charged the wrong amount.

Recover. Honour or correct affected carts (a product decision); the replayed DELs fix the cache.

Prevent.

• For a pure cache, turn snapshots off: restarting empty and warming (Break 3) is safer than restarting wrong. If you keep them, make "Redis restarted" trigger the replay job automatically.
• Send invalidations through a durable log instead of a fire-and-forget DEL: the write API writes an outbox row in the same transaction as the update; an invalidator consumes it and retries DELs until they land. When Redis is down, the consumer's backlog grows and is replayed when Redis returns. No DEL is lost.
• Put a version in every cached value so a sampler (and the follow-up below) can compare.

A race between a cache fill and an invalidation, made likely by replica lag. Request A misses at t = 0 and reads product 8812 from a replica that is 500 ms behind: it gets the old price. At t = 100 ms the writer commits the new price on the primary and DELs the key, which is already absent, so the DEL does nothing. At t = 200 ms request A SETs the old price it read. The cache now holds the old price with a fresh 1 h TTL, and no further DEL is coming.

Fixes, cheapest first: delete twice: DEL on write, then DEL again after the maximum replica lag (say 2 s, from a delayed job). Or version the value: the fill writes {version, data} with a small Lua script that refuses to overwrite a newer version, and the writer SETs the new version instead of deleting. Or fill misses for recently written keys from the primary. Then put a bound on it: alert when replica lag passes 1 s, because every one of these fixes assumes the lag is bounded (Chapter 5).

What the sim shows. Adding two shards left the busiest shard at 127% of capacity: the hot key alone was 96% of a shard, and more shards only thinned out the keys next to it. Turning on the cache in front of that key fixed it. Same here: 150,000 reads/s on one key is 150% of one Redis node, and three more shards leave that key exactly where it is.

Detect. Per-node CPU and ops/s (one node far above the others is the signature); a sampled top-K of keys in the client, or redis-cli --hotkeys (it needs an LFU eviction policy). Know which key it is within minutes, because every fix is per key.

Contain. Put the hot key in L1 with a 1 s TTL and single-flight per pod. Each pod then reads it from Redis once per second: with 40 pods (autoscaled for the launch), Redis sees 40 reads/s for that key instead of 150,000. Stock is up to 1 s stale on the product page, which matches the 2 s polling.

Degrade. If even the pods are overloaded, serve the stock endpoint from the CDN with Cache-Control: max-age=1. Every edge location then asks the origin about once per second, and the origin sees a few hundred requests per second at most.

Recover. Nothing to recover once the reads are absorbed; the node's CPU drops at once.

Prevent. For known launches, flag the product as hot ahead of time (pin it in L1, CDN-cache its stock endpoint). For unknown ones, automatic hot-key promotion: the client's top-K sampler moves any key above, say, 5,000 reads/s into L1. The other tool is key replication: write 8 copies, product:9001#0 to #7, which hash to different nodes, and read a random one. It spreads reads but makes every invalidation 8 DELs. Better still, replace polling with a push channel (SSE) that sends one message when stock changes.

Every UPDATE products SET stock = stock - 1 WHERE id = 9001 takes the same row lock and holds it until commit. If each transaction holds the lock for ~2 ms (statement plus commit flush), one row does at most about 500 updates/s. At 4,000/s the lock queue grows by 3,500 per second, connections fill with waiters, and checkout for every product slows because the pool is shared.

Fixes: split the stock into buckets: 20 rows of 500 units each, and each purchase decrements a random bucket that still has stock (WHERE stock > 0), giving 20× the write rate. The Hot write key tab of the hot-shard sim shows the same idea with salted sub-keys and its price: reading the total now fans out to every bucket. Or move the hot counter to Redis (DECR is atomic and one node does far more than 4,000/s), with the order written to Postgres afterwards and a reconciliation job. Or put purchases in a queue and let one consumer allocate stock in order, which turns contention into a waiting line (Drill 12 covers queues).