Part 3 · Demand

Reliability, Scalability, SLOs, and Tail Latency

Lesson 05 kept a record's meaning stable as code changes underneath it — a contract across time. Now real users arrive, and the system must answer to demand: load, latency, and the promises you make under both. This lesson defines the yardstick the whole track is measured against — the three non-functional properties every design must keep true: what stays correct when components fail, what stays fast enough when load grows, and what stays changeable when humans keep editing the software. Every later mechanism — replication, partitioning, quorums, streams — is a move that buys one of these and bills another. You cannot grade a move until you can name what it protects.

Book source

Original synthesis drawing its arc from Martin Kleppmann, Designing Data-Intensive Applications, Chapter 1 ("Reliable, Scalable, and Maintainable Applications"). The prose, numbers, and the tail-latency widget below are ours; the three-property framing is DDIA's.

Linear position

Prerequisite: Lesson 00 (Orientation) — the idea that every data system is a position in one design space, and the habit of naming the constraint before naming the tool.
New capability: A vocabulary for the three constraints themselves. After this you can state a load with concrete parameters, read a latency distribution in percentiles instead of averages, predict how tail latency amplifies under fan-out, and judge any design by which promise it protects and which it taxes.

The plan

Four moves. (1) Reliability — separate a fault from a failure, walk the three fault classes (hardware, software, human), and show why the cure is redundancy plus deliberately inducing faults. (2) Scalability — refuse to discuss scale until load is described with load parameters; then teach percentiles properly (p50/p95/p99/p99.9), show why the mean hides the tail, and derive tail-latency amplification with a worked number and an interactive widget. (3) Separate throughput from response time, and vertical from horizontal scaling. (4) Maintainability — operability, simplicity, evolvability — the bill a human pays on a bad Tuesday. Then failure modes, a checklist, and the hand-off into data models.

a design choice | +------------------------+------------------------+ | | | RELIABILITY SCALABILITY MAINTAINABILITY correct despite fast enough as changeable as faults load grows humans edit it | | | faults vs failures load params + percentiles operability redundancy throughput vs latency + simplicity induce faults on vertical vs horizontal + evolvability purpose | | | every later mechanism is judged by which of these it buys and which it bills.

1 · Reliability: correct behavior despite faults

Reliability is the promise that the system keeps doing what the user expects even when things go wrong. To make that precise we need two words people routinely confuse.

A fault is one component deviating from its spec: a disk returns a corrupt block, a process pauses for two seconds in garbage collection, a network packet is dropped, a deploy ships a logic bug, an operator runs DROP TABLE on the wrong host. A failure is when the system as a whole stops providing its required service to the user. The entire discipline of reliability is the art of preventing faults from cascading into failures. A fault is local and expected; a failure is global and is the thing you actually promised not to do. A system that tolerates faults can experience a thousand of them a day and never fail.

Faults come in three classes, and each demands a different defense.

Hardware faults

Disks die (a single disk's mean-time-to-failure is ~10–50 years, so in a 10,000-disk fleet you lose roughly one a day), RAM bit-flips, a rack loses power, a network link drops. Random and largely independent. Cure: redundancy — RAID, dual power, replicas on separate machines so one death is invisible.

Software faults

A bug triggered by a specific input, a memory leak, a runaway process that pegs a shared resource, a cascading retry storm. Correlated — they hit every node running the same code at once, so redundancy does not save you. Cure: testing, isolation, process restarts, measuring assumptions, and careful deploys.

Human faults

Operators misconfigure, mis-deploy, fat-finger a command. The largest cause of outages in practice. Cure: well-designed abstractions and APIs that make the right thing easy, sandboxes and staging, fast and reversible rollbacks, and detailed telemetry so a mistake is seen before it spreads.

The key asymmetry: redundancy defeats the independent faults (hardware) but is useless against the correlated ones (a bug deployed everywhere). That is why reliable systems do more than keep spare copies — they build idempotent operations so retries do not double-apply work, keep a durable log before acknowledging a write so a crash mid-operation is recoverable (we built that log in lesson 03), and expose enough observability that a human sees partial failure before it becomes corruption.

Counter-intuitive move: induce faults on purpose

You only know a fault-tolerance mechanism works if it has fired. A failover path that has never executed is not a safety feature; it is untested code that runs for the first time during your worst incident. So mature teams deliberately inject faults — kill a random node, drop packets, add 200 ms of latency, revoke a disk — continuously, in production. (Netflix's "Chaos Monkey" is the canonical example.) Triggering faults on a calm Tuesday converts a rare untested path into a routine, exercised one, and surfaces the correlated software faults that redundancy alone hides. Reliability is not the absence of faults; it is the rehearsed survival of them.

ML framing. In ML infrastructure the same logic governs the model registry and training-data lineage. A failed feature-transform job should not silently emit a half-empty training set (a fault becoming silent data corruption). A model promotion should be atomic — it either fully happens or not at all — so a crash never leaves the serving tier pointing at new weights while still feeding them yesterday's feature schema. The promotion writes its intent to a durable log first, then flips the pointer; a crash replays cleanly.

2 · Scalability: describe the load before you solve it

Scalability is the system's ability to cope with increased load. The word is meaningless on its own — "is X scalable?" is a non-question — until you answer two sub-questions concretely: what is the load? and what happens to performance when it grows?

Load is described with load parameters: the handful of numbers that characterize what the system is being asked to do. Which numbers matter is workload-specific, and picking the right parameter is half the problem. The social home timeline is the canonical worked example this whole topic hangs on — DDIA 2e uses it to ground load, latency, and scalability, and so do we. The naive load parameter is "tweets posted per second," but the one that actually sizes the system is the fan-out: each post must be delivered to all of an author's followers, and a celebrity with 30 million followers turns one write into 30 million timeline inserts. The system that looks easy under "posts/sec" is brutal under "timeline deliveries/sec" — so the design (push to followers' timelines on write, vs. pull from authors on read, vs. a hybrid that pulls only for celebrities) is chosen by which load parameter dominates. We carry this example through percentiles and fan-out below; the full end-to-end design — read/write paths, the fan-out hybrid, the SLOs — is the worked case at lesson 29 (Case: Social Home Timeline).

The same lesson holds for a model-serving endpoint, whose load parameters might be: requests per second, the read/write ratio, the fan-out per request (how many downstream feature lookups one inference triggers), the cache hit rate, and the freshness SLA on features. The same nominal QPS can be trivial or brutal depending on the parameters: 20,000 req/s where each request reads one cached row is easy; 20,000 req/s where each request fans out to 100 feature lookups across a sharded store is a very different machine.

one inference request, fan-out = 5 client --> model endpoint | (must wait for ALL feature lookups to return) +--------+--------+--------+--------+ v v v v v feat-A feat-B feat-C feat-D feat-E <- 5 backend lookups | | | | | (whichever one is SLOWEST sets this request's latency)

Percentiles, properly

Once load is named, performance is described as a distribution of response times — and you read that distribution at percentiles, never as a mean. A percentile pNN is the value below which NN% of requests fall. Sort every request's latency from a window and:

p50 (the median): half of requests are faster than this. The "typical" experience.
p95: 95% are faster, 1 in 20 is slower. The edge of the common case.
p99: 1 in 100 requests is slower than this.
p99.9: 1 in 1,000 requests is slower. The "tail."

Why not the mean? Because the mean is dominated by, and yet hides, the tail. Consider ten requests with latencies, in milliseconds: 10, 10, 12, 12, 13, 14, 15, 16, 18, 900. The mean is (10+10+12+12+13+14+15+16+18+900) / 10 = 102 ms — a number describing no actual request: nine were near 13 ms and one was catastrophic. The p50 is 13 ms (the honest "typical"), and the p90 is 900 ms (the honest "worst common case"). A single average of 102 ms would let you believe the system is mediocre-but-uniform when in truth it is excellent-but-occasionally-broken. The tail is where queueing, GC pauses, retries, TCP retransmits, and overloaded partitions reveal themselves — and the tail is what your most active users hit, because the more requests a user makes, the more likely one lands in the slow bucket.

The mean is the wrong summary, twice over

First, it is not robust: one 900 ms outlier drags the average far from any real request. Second, it gives you no guarantee to put in an SLA — "average 100 ms" permits 1% of requests to take 10 seconds. SLAs are written on percentiles for exactly this reason: "p99 < 200 ms" is a promise about the worst 1 in 100, which is a promise you can be held to.

Tail-latency amplification: the killer of fan-out systems

Now the move that makes tail latency a systems problem and not just a metrics nicety. Suppose a request fans out to N backends and must wait for all of them before it can respond (the model endpoint above — it cannot rank until every feature has arrived). Suppose every backend is healthy and independently meets p99 = 10 ms: each backend is "fast" (under 10 ms) with probability 0.99 and "slow" on the unlucky 1%.

The parent request is slow if any one of its N backends is slow. With independent backends, the probability that all N are fast is 0.99 ^ N, so:

P(request is slow) = 1 - 0.99 ^ N N = 1 : 1 - 0.99^1 = 1.0% (just the backend's own p99) N = 10 : 1 - 0.99^10 = 9.6% N = 50 : 1 - 0.99^50 = 39.5% N = 100 : 1 - 0.99^100 = 63.4% <-- a MAJORITY of requests now hit a slow backend N = 200 : 1 - 0.99^200 = 86.6%

Read N = 100 carefully: a 1%-rare event at each backend has become a 63% chance at the parent. The "p99 of one backend" is now roughly the p37 of the fanned-out request — the slow case stopped being rare and became the common case. This is tail-latency amplification: fanning a request out to many backends multiplies the chance of touching at least one slow one, so the parent's tail is far worse than any child's tail. Worse, your busiest users (highest QPS, highest fan-out) are exactly the ones who feel it most.

Worked number — what p99 must each backend hit?

Invert it. If you want the parent request to keep its own p99 — slow no more than 1% of the time — with fan-out N = 100, you need p ^ 100 = 0.99, where p is each backend's probability of being fast. Solving: p = 0.99 ^ (1 / 100) = 0.9998995, i.e. each backend must be slow no more than ~0.01% of the time. In percentile terms, a parent that fans out to 100 backends and wants its own p99 requires each backend to hit roughly p99.99. Tightening the tail by two extra nines is brutally expensive — which is why fan-out systems instead attack N (cache to skip lookups, batch many lookups into one call) or hedge (fire a duplicate request to a second replica after a short delay and take the first to answer, capping the tail at the timeout rather than the slowest backend).

Tail-latency amplification — feel the fan-out

Each backend independently hits its target with probability p99 = 99% (slow 1% of the time). Drag the fan-out N — the number of backends one request must wait on — and watch how fast the chance that the parent request hits at least one slow backend climbs. The bar fills with the fraction of requests that go slow; the readouts show the parent's effective slow-rate and the p99 each backend would need so the parent could still hold its own p99.

fan-out N: 100 each backend slow-rate: 1.0%

P(request hits a slow backend)

—

Effective parent percentile kept fast

—

Each backend needs (to keep parent p99)

—

Show the core JS

// each backend is "fast" with prob f = 1 - slowRate; parent waits for ALL N.
const f = 1 - slowRate;                  // e.g. 0.99
const pAllFast = Math.pow(f, N);         // 0.99 ^ N
const pSlow    = 1 - pAllFast;           // chance parent hits >=1 slow backend
// to keep the PARENT slow <= 1% (its own p99), each backend must be fast at:
const needFast = Math.pow(0.99, 1 / N);  // e.g. 0.99^(1/100) = 0.99990

3 · Throughput vs response time; vertical vs horizontal

Two pairs of words that are easy to slur together and costly to confuse.

Response time is what a single client waits — the time from sending a request to receiving the answer, including queueing. Throughput is how many requests the system completes per unit time. They are not the same dial, and they trade against each other: batching ten lookups into one call raises throughput (fewer round trips, better amortization) but can raise the response time of the first request in the batch (it waits for the batch to fill). A system tuned for maximum throughput (large batches, full queues) often has terrible tail latency, because a full queue means every new request waits behind a long line.

Why utilization near 100% is a trap

Response time does not degrade linearly with load — it degrades hyperbolically near saturation. As utilization approaches 100%, each new request finds fewer idle gaps to slip into, so queueing time explodes. A system can look healthy at 50% utilization and fall over at 90% with no single request getting more expensive — the cost is all in the wait. This is why an SLO must name both a percentile target and the headroom it holds: "p99 < 200 ms at 60% utilization" is a sturdy promise; "p99 < 200 ms while saturated" is a fantasy. Run hot and the tail (and thus the fan-out amplification above) gets dramatically worse.

When load outgrows one machine, you scale one of two ways. Vertical scaling (scaling up) means a bigger machine — more CPU, RAM, faster disk. It is operationally simple (one node, strong local transactions, no distributed coordination) but hits a ceiling: the largest machine money can buy, and its cost grows super-linearly. Horizontal scaling (scaling out) means more machines sharing the load. It has effectively no ceiling, but it imports every hard problem in this track — replication (lesson 08), partitioning (lesson 11), and the consistency questions that follow — because state now lives in more than one place and those places can disagree. The honest rule: scale up until it hurts, scale out when you must, and know that scaling out is what turns the rest of this curriculum on.

4 · Maintainability: the bill a human pays

Maintainability is not the soft sibling of the other two. Most of the cost of software is not the initial build; it is the ongoing care — fixing faults, keeping it running, adapting to new requirements, paying down complexity. Maintainability is the property that decides whether the system can keep changing without collapsing under its own special cases. DDIA breaks it into three:

Sub-property	What it means	What erodes it
Operability	People can keep the system running smoothly and understand what it is doing: dashboards, logs, runbooks, predictable behavior, clear ownership, the ability to backfill and replay.	Opaque state, no telemetry, manual recovery steps, surprises during deploys.
Simplicity	The design has fewer states a new engineer must hold in their head. Managing complexity, not raw line count — removing accidental complexity that the problem did not require.	Tangled special cases, leaky abstractions, a distributed-transaction protocol that adds stuck-coordinator states to reason about.
Evolvability	Schemas, APIs, encodings, and derived views can change while old and new code coexist. The system bends to new requirements instead of fighting them.	Rigid schemas with no versioning, no backward/forward compatibility (the home of lesson 05), tightly coupled services.

The recurring trade-off this track will hammer: a design can buy scale by adding copies and asynchronous workflows, but every copy is another thing that can be stale, broken, or forgotten during a migration, and every async hop is another failure mode (lag, duplicates, replays, poison messages). Maintainability is the question of whether the on-call engineer can actually operate that design at 3 a.m. on a bad Tuesday — which is the only time it truly matters.

Failure modes

Treating redundancy as a cure-all. Replicas defend against independent hardware faults but are useless against a bug deployed to every node at once. Correlated faults need testing, canaries, and rollback — not more copies.
Reporting the mean. An average latency hides the tail entirely and gives no enforceable guarantee. One outlier moves the mean off every real request. Always quote percentiles.
Ignoring fan-out amplification. Shipping a per-backend p99 SLA while the user-facing request fans out to 100 of them — the parent's tail is far worse than any child's, and grows with N.
Running near saturation. Sizing for average utilization with no headroom, then watching queueing latency explode the first time traffic spikes.
Untested failover. A recovery path that has never run is not a feature; its first execution is during your incident. If you never induce faults, you never learn it is broken.
Buying scale with un-operable complexity. Adding async copies and services until no human can reason about the system's states, trading a throughput win for a maintainability collapse.

Decision checklist

Which faults are independent (redundancy helps) vs correlated (it does not)? Name a defense for each class.
Have you written down the load parameters — QPS, read/write ratio, fan-out, cache hit rate, freshness SLA — before discussing scale?
Is the latency SLA a percentile (p99/p99.9) at a named utilization, never a mean?
What is the fan-out N of a user-facing request, and what per-backend percentile does that force?
What headroom are you running with, and does the tail stay bounded under a 2× traffic spike?
Do you induce faults on purpose so the failover path is exercised before an incident needs it?
For each new copy or async hop you add for scale: who operates it, how does it recover, and how does a human see it go stale?

Checkpoint exercise

Try it

You are designing a model-serving endpoint at 20,000 req/s. Each inference fans out to feature lookups across a sharded feature store. (a) List the load parameters you would write down. (b) The product wants a user-facing p99 < 50 ms. If each feature lookup independently hits p99 = 40 ms and an inference fans out to 30 of them, estimate the fraction of requests that miss the budget, and state what per-lookup percentile would be needed to hold the parent's p99. (c) Name two changes that attack N instead of chasing extra nines, and say which reliability and maintainability bills each one sends.

Where is truth?

For reliability and scalability themselves, the system of record is the written SLO — the percentile target at a named utilization plus the load parameters it assumes (e.g., "p99 < 200 ms at 60% utilization, fan-out N, 8k req/s"); that document, not a dashboard's mood, is the contract. The copies / derived views are the latency histograms, error budgets, and dashboards computed from request logs. The freshness budget is the measurement window and the headroom you hold before saturation. The owner is the on-call team that answers when the budget burns. The deletion path is retiring or renegotiating an SLO that the workload outgrew. The reconciliation/repair path is the incident response and the deliberate fault injection that proves the failover works. The evidence it is correct is percentile measurements under real load, chaos-test results, and a burn-rate alert that fires before users feel it.

Where this points next

We now have the yardstick: reliability (correct despite faults), scalability (fast enough as load grows, measured in percentiles and stressed by fan-out), and maintainability (changeable on a bad Tuesday). Every choice from here is graded by which of these it protects and which it taxes. But naming the demand is not meeting it — the home timeline still has to answer reads in single-digit milliseconds. Lesson 07 takes the first concrete swing at demand: the indexes, caches, and serving paths that make a read cheap enough to hit its SLO. We will see how an index trades write cost for read speed, how a cache shifts the load parameters (hit rate becomes the dial that decides whether 20,000 req/s is trivial or brutal), and how the serving tier holds a tail-latency budget under fan-out.

Takeaway

Three properties govern every later choice. Reliability is preventing local faults from becoming a system-wide failure; redundancy beats independent (hardware) faults but not correlated (software/human) ones, so you also need idempotency, durable logs, observability, and the discipline of inducing faults to keep failover paths rehearsed. Scalability starts by naming load parameters, then describes performance in percentiles (p50/p95/p99/p99.9) because the mean hides the tail and gives no enforceable promise; under fan-out, tail latency amplifies — a 1%-slow backend becomes a 63%-slow parent at N=100, so a parent's p99 demands each backend hit p99.99, which is why you attack N or hedge rather than chase nines. Distinguish throughput from response time, and scale up until it hurts before scaling out (which turns on the rest of this track). Maintainability — operability, simplicity, evolvability — is the bill a human pays to keep the thing alive while it changes. Grade every mechanism by which promise it buys and which it bills.

Interview prompts

What is the difference between a fault and a failure, and why does it matter? (§1 — a fault is one component deviating from spec; a failure is the whole system stopping its promised service. Reliability is preventing faults from cascading into failures, so you design assuming faults are normal.)
Name the three fault classes and which one redundancy does not fix. (§1 — hardware faults are independent and cured by redundancy; software faults are correlated across all nodes running the same code, so redundancy fails — testing/isolation/canaries do; human faults are the biggest cause, cured by good abstractions, sandboxes, and fast rollback.)
Why report p99 instead of the mean latency? (§2 — the mean is dragged off every real request by outliers and gives no enforceable guarantee; a percentile names the worst 1-in-100 you can write into an SLA, and the tail is where queueing/GC/retries and your busiest users live.)
Explain tail-latency amplification with a number. (§2 — if each of N backends is slow 1% of the time and the parent waits for all, P(slow) = 1 - 0.99^N: 9.6% at N=10, 63.4% at N=100. The rare backend tail becomes the common parent case.)
A request fans out to 100 backends; what p99 must each hit to keep the parent at p99? (§2 — solve p^100 = 0.99 → p = 0.99^(1/100) ≈ 0.9999, i.e. roughly p99.99 per backend. Chasing nines is expensive, so reduce N (cache/batch) or hedge to a second replica instead.)
When do you scale vertically vs horizontally? (§3 — scale up for operational simplicity until you hit the biggest/most-expensive machine; scale out when you must, accepting replication, partitioning, and consistency — the rest of this track — because state now lives in multiple places that can disagree.)
Why is running at 90% utilization risky if 50% looks fine? (§3 — response time degrades hyperbolically near saturation: queueing time explodes as idle gaps vanish, so the tail blows up without any single request costing more. SLOs must name a percentile at a named utilization with headroom.)