Caching, the read-path lever
Most large systems are read-heavy — 100:1 reads-to-writes is unremarkable. A cache is a smaller, faster, second copy of hot data placed closer to the reader. It buys enormous latency and backend-load reduction; it costs you a second copy that can go stale, plus a new failure mode. Both of Karlton's hard things — cache invalidation and naming — bite here.
If your read:write ratio is R:1 and reads are repetitive (a Zipfian "hot set" — a skewed popularity distribution where a handful of keys get most of the traffic), then a copy of the hot set near the reader serves nearly all traffic. The lever is the hit ratio h: every read you serve from cache is a read the backend never sees and a round-trip the user never waits for. The entire lesson is learning that the value of h is wildly non-linear — the last point of hit rate is worth 10× the first.
Where caches live: a latency ladder
A cache is defined by where it sits, and every tier trades shared-ness against speed. Closer to the reader is faster but less shared (more copies to keep coherent); closer to the data is slower but authoritative. Re-using the latency numbers from lesson 02:
| Tier | Typical latency | Shared? | Buys / Costs |
|---|---|---|---|
| Client / browser cache | ~0 (local) | Per-user | Free round-trip elimination / you can't invalidate it — TTL only |
| CDN / edge | ~10–50 ms (geo-local) | Per-PoP | Offloads static + cacheable dynamic, geo-near / weak control, purge lag |
| API gateway / reverse proxy | ~1 ms | Per-DC | Centralized response cache / one more hop to manage |
| In-process / app-local | ~100 ns | No (per-node) | Nanosecond reads / N incoherent copies — coherence problem |
| Distributed cache (Redis/Memcached) | ~0.5 ms (1 net hop) | Yes | Shared, coherent-ish, huge / a network hop + a dependency that can fail |
| DB buffer pool / page cache | ~0.1 ms (RAM in DB) | Per-DB | Free, automatic / still behind the DB connection + query parse/plan |
The browser and CDN tiers don't expose a delete button — they obey a contract carried in HTTP headers, and it's worth knowing the mechanics. The origin stamps each response with a Cache-Control header that sets the rules: max-age=300 says "this is fresh for 300 seconds, serve it from cache without asking", while no-cache says "you may store it, but revalidate with me before every reuse". Once an entry goes stale, the cache doesn't blindly re-download — it makes a conditional request. The origin's earlier response included a validator, an ETag (a short fingerprint of the body); the cache echoes it back in an If-None-Match header, and if the content hasn't changed the origin replies 304 Not Modified with an empty body. That revalidates the entry cheaply — a tiny round-trip instead of re-sending the whole payload — which is the whole point of the contract. (See 03 for how the same headers shape your API responses.)
The in-process tier is seductive — 100 ns, no network — but it is per-node: with 50 app servers you have 50 copies, and an invalidation must reach all of them or you serve stale. That coherence problem is why a shared distributed tier (0.5 ms) often wins despite being 5000× slower per read: it has one copy to invalidate. The senior move is frequently a two-level (L1 in-process + L2 Redis) cache with short L1 TTLs to bound incoherence.
Read / write patterns
How data enters and leaves the cache relative to the database. Pick by your read:write ratio, your tolerance for stale reads, and whether a crash that loses un-flushed writes is acceptable.
| Pattern | Mechanic | Buys | Costs / risk |
|---|---|---|---|
| Cache-aside (lazy) | App reads cache; on miss loads DB, then populates cache | Simple, resilient (cache down ≠ outage), only caches what's read | First hit is slow (cold miss); stale window between DB write and key expiry/invalidation |
| Read-through | Cache library itself loads from DB on miss | App code is cache-agnostic; centralized load logic | Couples you to the cache lib; same cold-miss penalty |
| Write-through | Write cache + DB synchronously | Cache always fresh; reads never stale post-write | Slower writes (two stores, both on the critical path) |
| Write-back (write-behind) | Write cache, async flush to DB | Very fast writes; absorbs write bursts; coalesces | Data loss on crash before flush; ordering/consistency complexity |
| Write-around | Write DB only; cache fills lazily on later read | Avoids caching write-only data that's never re-read | Recently-written data is a guaranteed miss (read-after-write is slow) |
The default for a generic read-heavy service is cache-aside + write-around with a TTL: it is the most resilient (a cache failure degrades latency, not correctness) and only caches data someone actually reads. Reach for write-through when read-after-write freshness matters; write-back only when write throughput is the bottleneck and you can tolerate a bounded loss window (or back it with a durable log).
The read flow and the economics
Two formulas carry the entire chapter. With hit ratio h, cache latency t_cache, and miss (backend) latency t_miss:
effective latency ≈ h · t_cache + (1 − h) · t_miss
backend load = (1 − h) · QPS
The backend-load formula is where the non-linearity hides. The miss rate is (1 − h), and it falls geometrically as h approaches 1:
| Hit ratio h | Miss rate (1−h) | Backend QPS @ 50k | Backend load vs h=0.90 |
|---|---|---|---|
| 0.90 | 0.10 | 5,000 | 1× (baseline) |
| 0.99 | 0.01 | 500 | 10× less |
| 0.999 | 0.001 | 50 | 100× less |
Going 90% → 99% cuts backend traffic 10×; 99% → 99.9% cuts it another 10×. The last few points of hit rate are worth the most. This is why teams obsess over warming, key design, and TTL tuning long after the cache "works" — they're chasing the geometric tail.
Service at 50,000 QPS, t_cache = 0.5 ms, t_miss = 20 ms, hit ratio h = 0.95.
- Effective latency ≈ 0.95 · 0.5 + 0.05 · 20 = 0.475 + 1.0 = ≈ 1.48 ms. (Without cache it would be a flat 20 ms — a ~13× win, and the misses dominate the average even at 5%.)
- Backend load = (1 − 0.95) · 50,000 = 2,500 QPS. The DB sees 1/20th of the traffic.
- Push to h = 0.99: backend = 0.01 · 50,000 = 500 QPS (5× less DB load for +4 points), effective ≈ 0.99·0.5 + 0.01·20 = ≈ 0.695 ms.
- Capacity rule: if the cache tier fails, h → 0 and the backend instantly faces the full 50,000 QPS — 20× its steady-state. Your DB must either survive that, or you must shed/degrade. Never size the backend for the cached case alone.
Eviction & working-set fit
A cache is finite; eviction decides what to keep. The policy only matters if your working set doesn't fit — if it does, everything stays and any policy gives ~100% hits.
- LRU (least-recently-used): evict the coldest. Default; matches temporal locality. Vulnerable to a large scan flushing the hot set.
- LFU (least-frequently-used): keep the genuinely popular. Scan-resistant; needs decay or it ossifies on yesterday's hot keys.
- FIFO: evict oldest inserted. Cheap, ignores access — usually worse than LRU.
- TTL: time-bounded, orthogonal to the above. Bounds staleness and forces re-fetch even of hot keys.
The economics: hit rate is a function of cache size ÷ working-set size. Because access is Zipfian, the curve is concave — a small cache catches the head cheaply, then you pay a lot of RAM for each additional point. That concavity is the dual of the backend-load non-linearity above: the marginal point of h costs the most RAM and saves the most DB.
Consistency & invalidation
A second copy can disagree with the source of truth. Two strategies, both imperfect:
- TTL expiry: data is correct within a bounded window of at most the TTL. Dead simple, no write-path coupling — but you accept stale reads up to TTL and pay a re-fetch on every expiry. Good for tolerant data (catalog, profiles).
- Explicit invalidation on write: delete/update the key whenever the DB changes. Fresh, but every write path must remember to invalidate — miss one and you serve stale forever. Hard to get right, which is why it's the "hard thing."
- Versioned keys: embed a version/etag in the key (
user:42:v7). A write bumps the version; old keys are simply never read again and age out by TTL. Sidesteps the delete-everywhere race at the cost of key churn.
The honest framing in an interview: "How stale can a read be, and what does that cost the business?" If the answer is "seconds are fine," TTL wins on simplicity. If it's "must be fresh," you pay the invalidation tax (write-through or explicit deletes).
Failure modes (and the fix for each)
A cache adds a new layer that fails in characteristic ways. The recurring shape is the thundering herd — many requests converging on one slow path (forward-ref lesson 13).
| Failure | What happens | Fix |
|---|---|---|
| Stampede / dogpile | A hot key expires; thousands of concurrent misses all hit the DB to recompute the same value | Single-flight / request-coalescing (one loader per key, others wait); serve-stale-while-revalidate; probabilistic early refresh (refresh before TTL with rising probability) |
| Penetration | Requests for keys that don't exist miss the cache every time and pound the DB (often malicious) | Negative caching (cache the "not found" with a short TTL); a Bloom filter of existing keys to reject impossible lookups before the DB |
| Avalanche | Many keys expire at the same instant, or the whole cache tier goes down → full QPS hits the DB at once | Jittered TTLs (spread expiry over a window); replicate the cache tier; circuit-break / degrade to a read-only or partial path so the DB survives h → 0 |
Setting TTL = 300s for a batch of keys loaded together (e.g. on deploy) guarantees they all expire together 5 minutes later — a self-inflicted avalanche. Always set TTL = base + random(0, jitter). The same jitter quietly defuses stampedes too, since hot keys no longer expire in lockstep. It is the single cheapest reliability fix in caching.
Hit-rate economics calculator
Move the hit ratio and watch backend QPS collapse geometrically. The "next point" KPI shows the marginal value of one more percentage point of h — the core senior intuition.
Interview prompts you should be ready for
- "Your read path is at p99 = 40 ms and the DB is the bottleneck. Where do you add a cache and which pattern?" (senior answer: read:write ratio first — if read-heavy, cache-aside + write-around at a shared Redis tier; quantify with effective-latency and backend-load formulas; pick the tier by the shared-vs-speed trade-off, and only go in-process L1 if I can tolerate/bound incoherence with a short TTL.)
- "We're at 95% hit rate. Is it worth engineering effort to reach 99%?" (senior answer: yes — backend load drops from (1−0.95)=5% to 1% of QPS, a 5× reduction in DB load and a real latency win, because the miss term dominates the average. The marginal point of h is the most valuable; show the geometric table.)
- "A single hot key expires and the DB falls over. What happened and how do you prevent it?" (senior answer: cache stampede — N concurrent misses recompute the same value. Fix with single-flight request coalescing, serve-stale-while-revalidate, and probabilistic early refresh; plus jittered TTLs so hot keys don't expire in lockstep.)
- "How do you keep the cache consistent with the database?" (senior answer: you don't, fully — you choose a bounded-staleness model. TTL for tolerant data; explicit invalidation or write-through where read-after-write freshness matters; versioned keys to dodge the delete-everywhere race. The question is always 'how stale is acceptable and what does it cost.')
- "Attacker sends requests for random non-existent IDs. Cache hit rate craters. Why and what do you do?" (senior answer: cache penetration — missing keys bypass the cache every time. Negative-cache the not-found with a short TTL and/or front it with a Bloom filter of valid keys so impossible lookups never reach the DB.)
- "What's your blast radius if the Redis tier goes down at peak?" (senior answer: h→0 instantly, so the backend faces full QPS — often 10–20× steady state. I must size/shard the DB to survive it, or degrade: shed load, serve stale, circuit-break, replicate the cache tier. Never size the backend for the cached case only.)
- "In-process cache vs distributed cache — when and why?" (senior answer: in-process is ~100 ns but per-node, so N incoherent copies and an invalidation that must fan out to all N; distributed is ~0.5 ms but one shared copy to invalidate. Two-level L1+L2 with short L1 TTL gets most of the speed while bounding incoherence.)
- "Write-through vs write-back — what would make you choose write-back?" (senior answer: write throughput is the bottleneck and a bounded loss window on crash is acceptable, or it's backed by a durable log. Write-back coalesces and absorbs bursts but trades durability and ordering simplicity; default is write-through/write-around unless writes are the proven hot spot.)
A cache is a faster second copy of hot data, closer to the reader. Its value is governed by two formulas — effective latency ≈ h·t_cache + (1−h)·t_miss and backend load = (1−h)·QPS — and the killer insight is that backend load falls geometrically in h, so the last points of hit rate are worth 10× the first. You pay for this with a second copy that can go stale (choose your bounded-staleness model: TTL, invalidation, or versioned keys) and a new failure tier (stampede, penetration, avalanche — each with a known fix, jittered TTLs being the cheapest). Above all: when the cache dies, h→0 and the full firehose hits the backend, so capacity must survive the un-cached case.