Part 12 · Applied & interviews
Failure Timelines and Quantitative Drills
Lesson 27, the production atlas, placed every real system at its coordinates in the design space the track built — storage engine, replication model, partitioning, consistency. Knowing where a system sits tells you what it promises when things go well. This lesson rehearses the other half of senior judgment: how those systems break, traced minute by minute, and how you size them before you build. The first half is six failure timelines you can narrate in an incident review; the second is eight quantitative drills — a named one-line formula and a worked number you can redo on a whiteboard. No new mechanisms here. This is a lab: ASCII timelines and arithmetic, wiring together the whole mechanism arc from L06 to L20.
New capability: Narrate a data-system incident as a time-ordered sequence of actors and events — name the root cause, the user-visible symptom, and the fix — and back-of-envelope the size of a system (QPS, partitions, storage, lag, backlog, replay) from a one-line formula with a worked number, the two things a senior interview probes hardest.
Part A — Six failure timelines
Every diagram below uses the same convention: each row is an actor (a process, a machine, a client), time runs left to right across columns t0…tN, and a marked cell is an event at that actor at that time. Read it like a sequence diagram drawn in fixed-width type. After each, three lines: the root cause (the mechanism that made it possible), the symptom (what a user or a dashboard sees), and the fix (the design change that closes it).
1 · Failover with lost writes
Single-leader replication (L08) with asynchronous replication: the leader acknowledges a write to the client as soon as it is durable locally, before followers have copied it. That ack is a promise. Failover can break it.
Root cause. Async replication means an acknowledged write can exist on the old leader alone. When that leader dies and a follower behind by even one entry is promoted, the unreplicated tail of the log is discarded — the new leader has no record of X, and the old leader's extra entries are thrown away when it rejoins.
Symptom. A client that got "200 OK, saved" reads back NOT FOUND seconds later. Worse if those positions were reused: when a newly promoted leader B reassigns an auto-increment id to a different row than the one the old leader gave it to, the lost write silently aliases someone else's data — a documented class of failover bug, and the reason auto-increment keys are dangerous across a leader change.
Fix. Trade latency for safety on the writes that need it: semi-synchronous replication (the leader waits for at least one follower's ack before acking the client, so any survivable single failure still holds the write), or a consensus-replicated log (L17) where a write is committed only once a quorum has it — then a promoted node provably holds every committed entry. For the autoincrement-aliasing variant, never reuse positions/ids across a failover (use a fencing token, L15).
2 · Stale read breaks read-your-writes after a leader change
Read-your-writes consistency (L08) — the guarantee that a user who just wrote sees their own write — is usually maintained by routing that user's reads to the leader (or to a follower known to be caught up). A leader change plus a stale routing cache breaks it, and it is distinct from §1: here no write is lost, it just hasn't propagated to the replica a confused client reads from.
Root cause. Two compounding facts. The routing layer's notion of "where the leader is" went stale at failover (a stale routing map, L11), and the replica it now points the client at is behind by the replication lag. The write is safe on B; the client simply isn't reading from B.
Symptom. "I just changed my photo and it reverted." Read-your-writes appears to work, then intermittently fails for ~one lag-window after any leader election — the most confusing kind of bug because it is timing-dependent and self-heals.
Fix. Make the session sticky to monotonic progress, not to a machine: track the log position (LSN / version) of the user's last write in their session, and on read either route to the leader or pick a follower whose applied position is ≥ that token (this is the bridge to monotonic reads and linearizability, L16). Refresh the routing map from the coordination service on election, do not cache leader identity past a lease.
3 · CDC consumer crash and offset backlog
A change-data-capture (CDC) stream (L19) reads the database's replication log as an ordered event stream and a downstream consumer maintains a derived view (a search index, a cache, a feature table). The consumer commits an offset — its position in the log — so it can resume. A crash plus the wrong commit discipline gives you either lost updates or a slow, visible backlog.
Root cause. The consumer committed its offset before the work was durable downstream (or only periodically), so on crash it resumes from an old offset. Meanwhile the producer never paused — the log is a buffer that keeps filling — so the backlog is the downtime times the produce rate, plus the re-read gap.
Symptom. The derived view is frozen at the crash point and then lags for minutes: search results go stale, the feature table serves old values, dashboards flatline and then "catch up" in a rush. If offsets were committed after processing, replayed events from e100..e140 are re-applied — fine only if the consumer is idempotent.
Fix. Commit offsets only after the downstream write is durable, and make every apply idempotent (upsert by primary key, or dedupe on an event id) so at-least-once replay is safe — this is exactly the log-driven discipline of L20. Size the consumer to drain faster than the producer (Drill 7 computes the catch-up time), and alarm on consumer lag, not just liveness.
4 · Rebalance storm
Partition placement (L11) computed as hash(key) mod N, where N is the node count. It looks balanced until N changes — then almost every key's home moves at once and the data movement starves live traffic.
Root cause. The node count N leaked into the placement function. Going from N=10 to N=11, only about 1/11 ≈ 9% of keys keep their home; roughly 91% must migrate (Drill: see L11's "hash mod N" callout). The migration competes with live reads and writes for disk and network bandwidth.
Symptom. Adding capacity tanks latency cluster-wide for the duration of the move — p99 spikes, timeouts, retries amplifying load — the opposite of what the operator intended.
Fix. Remove N from placement: a fixed number of partitions far larger than the node count (move whole partitions, ~1/N of them, when a node joins), consistent hashing (a key reassigns only the arc to its new neighbor), or dynamic partitioning (split/merge by size). All move ~1/N of the data instead of (N−1)/N. Independently, rate-limit rebalancing so migration never consumes more than a fixed slice of bandwidth, and prefer to rebalance during low-traffic windows.
5 · Cache stampede (thundering herd)
A read-through cache fronts an expensive query. A single popular key expires; every concurrent reader misses simultaneously and all of them hit the database at once to recompute the same value — the thundering herd. This is a tail-latency and load-amplification failure (L06), not a correctness one.
Root cause. A single expiry event is shared by every concurrent reader of that key, so a miss is not one recompute but N simultaneous recomputes of the identical value. Aligned TTLs make it worse: many keys expiring on the same boundary multiply the herd.
Symptom. Periodic DB load spikes synchronized to TTL boundaries; p99 sawtooths; under enough concurrency the recompute itself slows, the herd grows, and the cache never refills — a metastable collapse.
Fix. Three composable mitigations: request coalescing (single-flight) — the first miss takes a per-key lock and recomputes; concurrent missers wait for that one result instead of each hitting the DB; jittered TTL — set expiry to base ± random so keys do not expire in lockstep; and stale-while-revalidate — serve the expired value immediately while a single background task refreshes it, so a miss never blocks a reader. Combined, one key's expiry becomes one background recompute, not 5,000 foreground ones.
6 · Dual-write divergence
The application writes the same fact to two stores directly — the database and a search index (or cache) — in sequence, with no shared transaction (L20). Any partial failure or interleaving leaves the two permanently disagreeing.
Root cause. Two independent writes with no atomicity and no agreed ordering. The second write can fail while the first succeeds (partial failure), or two writers can apply to the two stores in opposite orders (race), and nothing ever reconciles them — there is no single source of truth that the index is derived from.
Symptom. The search index and the database silently disagree — a price, a status, a name shows one value on the listing and another on the detail page — and it never self-corrects. The longer it runs, the more keys drift.
Fix. Stop dual-writing. Make the database the single source of truth and derive the index from its change log: write once to the DB, capture the change via CDC / an outbox (L19), and stream it to the indexer with idempotent upserts (L20). Now ordering comes from the one log, a failed index write is just unconsumed offset (retried, never lost — see §3), and the index is eventually consistent by construction rather than by hope. This is the flagship lesson of the whole track, and the dedicated case study (L32) builds it end to end.
Part B — Eight quantitative drills
Each drill is a named one-line formula and a worked plug-in you can reproduce. These are the back-of-envelope numbers a senior interview expects you to do out loud. Keep the arithmetic round; the goal is the right order of magnitude and the right shape of the relationship, not three significant figures.
Drill 1 · QPS & p99 budget (Little's Law)
A service handles throughput = 50,000 QPS at mean latency = 20 ms = 0.020 s. In-flight requests = 50,000 × 0.020 = 1,000 concurrent. If each request pins one thread/connection, you need ~1,000 worker slots; at 200 threads you would queue, and queueing pushes latency up, which raises concurrency further — a feedback loop.
p99 budget split. Suppose the end-to-end p99 target is 200 ms and a request makes 4 sequential downstream calls plus 20 ms of its own compute. Budget left for downstreams = 200 − 20 = 180 ms, so each sequential hop gets 180 / 4 = 45 ms at p99. Any dependency whose p99 exceeds 45 ms blows the budget — that is the SLO you hand each team.
Drill 2 · Fan-out tail amplification
A scatter-gather query hits f = 100 partitions; each is slow (above your threshold) just p = 1% of the time. Chance at least one is slow = 1 − (0.99)^100 = 1 − 0.366 = 0.634. So 63% of these fan-out requests are slow even though each leaf is fast 99% of the time. At f = 200, 1 − (0.99)^200 = 1 − 0.134 = 0.866 — 87%. This is why a leaf's p99 becomes the parent's median, and why hedged requests, fewer leaves, or a tail-tolerant gather (return after the fastest 95%) matter.
Drill 3 · Partition-count sizing
Dataset = 4 TB, target ≤ 50 GB per partition (so each fits in cache/RAM budget and rebalances quickly): 4,000 GB / 50 GB = 80 partitions by size. Traffic = 200,000 QPS, a single node comfortably serves 5,000 QPS: 200,000 / 5,000 = 40 partitions by throughput. Take the max = 80 partitions. Then over-provision the partition count (e.g. round to 256) so you can add nodes by moving whole partitions without ever re-splitting — the fixed-partition rebalancing scheme from §4.
Drill 4 · Storage growth
10,000,000 rows/day × 2 KB/row × 3 replicas × 90 days. Step it: 10e6 × 2 KB = 20 GB/day of logical writes; × 3 = 60 GB/day on disk across replicas; × 90 = 5,400 GB ≈ 5.4 TB at steady state. Add headroom for compaction scratch and indexes (often 1.5–2×), so plan for ~8–11 TB. Daily growth of 60 GB/day also tells you the rebalance cadence: you cross the 50 GB-per-partition line (Drill 3) regularly, so dynamic splitting or a generous fixed count is mandatory.
Drill 5 · Compaction bandwidth
User writes arrive at 50 MB/s; a leveled LSM with ~10× write amplification rewrites each byte through the levels: background compaction I/O = 50 MB/s × 10 = 500 MB/s. If your disks sustain ~600 MB/s, compaction alone eats 83% of write bandwidth, leaving almost nothing for reads or for absorbing a write spike — the storage engine is the bottleneck, and the levers are a less aggressive compaction strategy (size-tiered trades read amp for less write amp) or faster storage.
Drill 6 · Replication lag
Leader takes writes at 80,000 rows/s; a follower (single-threaded apply, or busy serving reads) applies only 60,000 rows/s. Lag grows at 80,000 − 60,000 = 20,000 rows/s — after 60 s the follower is 1.2 million rows (~15 s of writes at 80,000/s) behind, and read-your-writes via that follower is broken for that window (the §2 bug). It only recovers when write rate falls below 60,000/s; if it never does, the follower is permanently stale and must be removed from the read pool or given parallel apply.
Drill 7 · CDC backlog & catch-up time
After the 30 s consumer outage in §3, the producer kept appending at 10,000 ev/s, so the backlog is 30 s × 10,000 = 300,000 events (plus the re-read gap; call it ~350,000). The restarted consumer drains at 40,000 ev/s while new events still arrive at 10,000 ev/s, so the net drain is 40,000 − 10,000 = 30,000 ev/s. Catch-up = 350,000 / 30,000 ≈ 12 s. If the consumer could only do 12,000 ev/s, net drain is 2,000 ev/s and catch-up is 350,000 / 2,000 = 175 s — and a consumer at ≤10,000 ev/s never catches up at all. Provision consume rate as a multiple of produce rate precisely to bound this.
Drill 8 · Replay / reprocessing time
You retain a 2 TB event log and reprocess at 200 MB/s (read + transform + index throughput of the rebuild job): 2,000,000 MB / 200 MB/s = 10,000 s ≈ 2.8 hours to rebuild from scratch. Parallelize across 10 partitions reading independently and it drops toward ~17 minutes (if the indexer scales linearly and partitions are even). This number is your recovery budget: it is how long a full rebuild of a corrupted derived store takes, and it argues for keeping derived stores cheap to rebuild rather than precious — the kappa-architecture bet of L20.
Failure → fix map
Timelines → root cause → fix
- 1 Lost writes on failover — async ack before replicate; promote a behind follower. → semi-sync or consensus-replicated log; fence ids (L08/L15/L17).
- 2 Stale read after election — stale routing map + follower lag. → version-token sticky reads, refresh map on election (L08/L16).
- 3 CDC backlog — offset committed too early; producer never pauses. → commit after durable + idempotent apply; size consumer faster than producer (L19/L20).
- 4 Rebalance storm — N in the placement function. → fixed partitions / consistent hashing / dynamic split; rate-limit migration (L11).
- 5 Cache stampede — shared expiry → N simultaneous recomputes. → request coalescing + jittered TTL + stale-while-revalidate (L06).
- 6 Dual-write divergence — two writes, no atomicity/order. → one source of truth, derive via CDC log, idempotent upsert (L20).
Sizing drills (one line each)
- 1 Little's Law — concurrency = throughput × latency; then split the p99 budget across hops.
- 2 Fan-out tail — P(slow) = 1 − (1−p)^f; the slowest leaf becomes the parent's tail.
- 3 Partitions — max(size/per-partition, QPS/per-partition); over-provision the count.
- 4 Storage — rows/day × bytes/row × replication × retention; add compaction headroom.
- 5 Compaction — write-rate × write-amp; watch it eat disk bandwidth.
- 6 Replication lag — write-rate − apply-rate; positive ⇒ unbounded staleness.
- 7 CDC catch-up — backlog / (consume − produce); the gap sets the time.
- 8 Replay — log-size / throughput; your derived-store rebuild budget.
Checkpoint exercise
Where this points next
You can now narrate how a data system breaks and size one before building it — the two skills the rest of Part IV exercises end to end. The next six lessons are full senior-interview case studies, each one a complete walkthrough that reuses exactly these timelines and drills: requirements and clarifying questions, a back-of-envelope sizing pass (Drills 1–8), a design built from the track's mechanisms, a concrete failure timeline (Part A), and a four-bucket answer rubric. Lesson 29 opens with the Twitter / social timeline — fan-out-on-write versus on-read, the celebrity hot-key problem (the §3/§4 hot-partition math made product-shaped), and the home-timeline cache (the §5 stampede in the wild).
Interview prompts
- A client got "200 OK" then reads NOT FOUND after a failover — what happened and how do you prevent it? (§1 — async replication acked before the write reached a follower; the promoted, slightly-behind follower discarded the unreplicated tail. Use semi-synchronous or consensus-replicated logs, and fence ids so positions aren't reused.)
- Read-your-writes works, then breaks for a few seconds after every leader election — why? (§2 — the routing map went stale and points reads at a follower behind by the replication lag; the write isn't lost, just not yet on the read replica. Fix with a version-token sticky read and refresh the map on election.)
- A CDC-fed search index froze, then "caught up" in a rush — diagnose. (§3, Drill 7 — the consumer crashed; the producer kept appending, so backlog = downtime × produce-rate. Catch-up = backlog / (consume − produce). Commit offsets after durable, apply idempotently, size consumer faster than producer.)
- Adding a node spiked p99 cluster-wide — what's the cause and the fix? (§4 — hash-mod-N put the node count in the placement function, so ~91% of keys migrated at once and starved live traffic. Use fixed partitions, consistent hashing, or dynamic split, and rate-limit migration.)
- One popular cache key expiring crushed the database — name the failure and three fixes. (§5 — cache stampede / thundering herd: every concurrent reader misses the shared expiry and recomputes the same value. Request coalescing (single-flight), jittered TTL, and stale-while-revalidate.)
- The search index and the database disagree on a price and never reconcile — root cause and fix? (§6 — dual write: two independent writes with no atomicity or shared order; a partial failure or race leaves them permanently split. Make the DB the single source of truth and derive the index from its change log with idempotent upserts.)
- A service does 50,000 QPS at 20 ms — how many requests are in flight, and how do you split a 200 ms p99 budget? (Drills 1 — Little's Law: 50,000 × 0.020 = 1,000 concurrent. Subtract own compute, divide the rest across sequential hops, e.g. 180 ms / 4 hops = 45 ms p99 each, the SLO per dependency.)
- A scatter-gather hits 100 partitions each slow 1% of the time — what fraction of queries are slow? (Drill 2 — 1 − 0.99^100 = 63%; the slowest leaf becomes the parent's tail, which is why you reduce fan-out, hedge, or gather after the fastest 95%.)