Latency, throughput & queueing
These three numbers are the measuring stick for every decision in the rest of the course. If you can't reason about them from first principles, every "add a cache / add a replica" answer is a guess. The headline you must internalise: latency and throughput are different quantities, they are often traded against each other, and a queue's response time blows up non-linearly as you approach full utilisation.
1. Latency and throughput are not the same number
Latency is the time to complete one request, measured in seconds (or ms). Throughput is the number of requests the system completes per second — its capacity. They answer different questions: latency is "how long does this user wait?", throughput is "how many users can I serve at once?".
The analogy: latency is the travel time for one car across the highway; throughput is the number of cars/hour the highway moves. Adding lanes raises throughput without making any single car faster. Raising the speed limit lowers per-car latency but does little for cars/hour if the on-ramp is the bottleneck.
The trap is assuming one implies the other. A microbenchmark that returns in 2 ms on an idle box tells you nothing about how many concurrent requests the box sustains — the moment requests queue, that 2 ms becomes 2 ms + wait. Conversely, a system can have huge throughput and terrible per-request latency: that's exactly what batching buys you.
2. The mean lies — you SLO on percentiles
Latency is not a number, it's a distribution, and that distribution is heavy-tailed (GC pauses, cache misses, lock contention, a slow disk). The mean is dominated by the bulk and hides the tail; the tail is where users actually suffer. So you specify and alert on percentiles: P50 (median — the typical user), P90, P99, P99.9 (the unlucky few). "P99 = 250 ms" means 99% of requests finished in ≤ 250 ms and 1% were worse.
| Statistic | Latency | What it tells you |
|---|---|---|
| Mean | 28 ms | Reassuring and useless — pulled down by the bulk |
| P50 | 20 ms | The typical request |
| P90 | 45 ms | Starting to feel the tail |
| P99 | 220 ms | The user who's actually angry |
| P99.9 | 1.4 s | Times out, retries, files a ticket |
Mean ≪ P99 here: reporting 28 ms would be a lie about the experience. And the tail is not a rounding error — at scale it is a large absolute number of people. 1M requests/day × 1% = 10,000 bad experiences every day; at P99.9 that's still 1,000 timeouts daily. You SLO on P99 (often P99.9 for paid/critical paths) precisely because that fraction maps to real, repeat-affected users.
3. Tail-at-scale: fan-out amplifies the tail
The reason tail latency dominates distributed design: a request that fans out to n backends and must wait for all of them sees the maximum of n latencies, not the average. The slowest leaf sets your response time.
Model it: suppose each leaf meets its threshold t with probability p. If the leaves are independent, the probability that all n are fast is
P(all ≤ t) = pⁿ
P(all 100 ≤ 10 ms) = 0.99¹⁰⁰ ≈ 0.366
So only ~37% of fan-out requests finish under 10 ms — ~63% exceed 10 ms even though every single backend "meets its P99." Your P99 leaf became roughly your P37 root. The aggregate behaves like the leaf's tail, not its median.Mitigations, and what each buys/costs:
- Hedged / backup requests — after a short delay (e.g. P95) send a duplicate to a second replica, take the first to answer. Buys: the max collapses toward the median. Costs: a few % extra load (cap hedges to ~5%).
- Fewer hops / shallower fan-out — n in the exponent is brutal; halving fan-out depth is worth more than shaving each leaf. Costs: coarser sharding, bigger per-leaf work.
- Parallel-with-timeout + partial results — return from the leaves that answered, degrade gracefully. Costs: completeness/correctness of the result.
4. Little's Law — concurrency, rate and time are one equation
For any stable system, the long-run average number of requests in the system equals the arrival rate times the average time each spends in it:
L = λ · W
L = concurrency (requests in flight), λ = arrival/completion rate (req/s), W = time in system (s). It's distribution-free — no assumptions about arrival pattern. Use it both directions:
Sizing a pool: concurrency = throughput × latency
You serve 2,000 req/s and each request occupies a worker for 50 ms. Then L = 2000 × 0.05 = 100 requests in flight on average. So your thread pool / connection pool / worker count must be ≥ 100 just to keep up — and you want headroom above the average. Set the DB connection pool below this and requests queue before the DB; set it far above and you risk overloading the DB. This single multiply is the backbone of capacity planning.
Inferring latency from observed load
Reverse it: you measure 300 in-flight requests at 1,500 req/s. Then W = L / λ = 300 / 1500 = 0.2 s = 200 ms average in-system time — useful when you can count concurrency (queue depth, active connections) but can't directly instrument latency.
5. Queueing & utilisation — the non-linear killer
Here is the most important shape in systems design. Model a single server as an M/M/1 queue (a single server with random "Poisson" arrivals — meaning arrivals are independent and "memoryless", no two coordinated — and random service times): Poisson arrivals at rate λ, exponential service at rate μ (so mean service time 1/μ). Define utilisation (the arrival rate divided by the service rate, λ/μ — the fraction of time the server is busy):
Before the formula, the intuition for why a queue blows up near saturation. Picture a single supermarket checkout. When customers trickle in well below the cashier's pace, gaps open up and the cashier idles — and those idle moments are exactly what let any small bunch-up drain away before the next customer arrives. But as arrivals creep up toward the rate the cashier can clear, the idle gaps vanish: there is no longer any slack to catch up in. Now a tiny burst — three people arriving together — has nowhere to be absorbed, so the line it creates never fully clears before the next burst lands. The backlog feeds on itself, and the average wait balloons toward infinity even though arrivals are still technically below capacity. The queue explodes not because the server is overwhelmed on average, but because it has run out of the idle time it needs to recover from normal randomness.
ρ = λ / μ
For a stable queue you need ρ < 1 (you can't sustainably arrive faster than you serve). The standard M/M/1 results:
W = 1 / (μ − λ) · L = ρ / (1 − ρ) · Lq = ρ² / (1 − ρ)
W is mean response time (queue + service), L mean number in system, Lq mean number waiting. The signature is the 1/(1−ρ) factor: response time is the unloaded service time multiplied by 1/(1−ρ). As ρ → 1 this explodes.
| ρ | 1/(1−ρ) — wait multiple | L (in system) | Reading |
|---|---|---|---|
| 0.50 | 2× | 1.0 | relaxed |
| 0.70 | 3.3× | 2.3 | fine, watch it |
| 0.80 | 5× | 4.0 | climbing |
| 0.90 | 10× | 9.0 | danger |
| 0.95 | 20× | 19 | do not |
| 0.99 | 100× | 99 | fire |
Going from ρ=0.5 to ρ=0.9 is only an 80% increase in load but a 5× increase in response time. From 0.9 to 0.99 — a mere 10% more load — response time grows another 10×. This is why you never run a queue or server at 95–100% utilisation: a tiny traffic spike or a brief service slowdown pushes ρ past 1 and the queue (and latency) diverges. And M/M/1 is the optimistic case — real service times have higher variance than exponential, and the C² variability factor (the squared coefficient of variation — a measure of how bursty or variable the workload is, with the steady exponential case being C²=1) in the general (M/G/1) formula makes the blow-up start even earlier. Plan to run hot paths around ρ ≈ 0.5–0.7 and keep the headroom for spikes.
6. Levers — which number each one actually moves
| Lever | Primary effect | Cost / trade |
|---|---|---|
| Cache / do less work | latency ↓ (and μ ↑, so ρ ↓) | staleness, invalidation, memory |
| Add replicas / shards | throughput ↑, lowers ρ per node | $, coordination, fan-out tail |
| Batching | throughput ↑ | per-request latency ↑ |
| Pipelining / async I/O | throughput ↑, frees workers | complexity, harder debugging |
| Reduce fan-out depth | tail ↓ (smaller n in pⁿ) | coarser sharding |
| Hedged requests | tail ↓ (max → median) | ~few % extra load |
Diagnose before you prescribe: a P50 problem is a "do less work" problem (caching, fewer hops, faster algorithm). A P99 problem is usually a queueing or tail problem (utilisation too high, GC, fan-out, a slow replica) — and throwing CPU at the median often won't touch it.
Try it: an M/M/1 queue calculator
Move λ toward μ and watch response time go vertical. This is the single most useful intuition pump in the course — internalise the cliff.
Interview prompts you should be ready for
- "Why do we measure P99 instead of average latency?" (senior model answer: latency is heavy-tailed, so the mean is dominated by the bulk and hides the slow requests users actually feel; P99 names the experience of the worst-served 1%, which at scale — 1M req/day × 1% = 10k people — is a large absolute population, often the same retry-amplified users. SLO on the percentile that maps to harm.)
- "A request fans out to 50 services, each with a 10 ms P99. What's the P99 of the whole request?" (senior model answer: it sees the max, not the average. With independence, P(all ≤ 10 ms) = 0.99⁵⁰ ≈ 0.605, so ~40% exceed 10 ms — your aggregate P99 is far worse than 10 ms; the root behaves like the leaf tail. Mitigate with hedged requests, fewer hops, and timeouts with partial results.)
- "How big should this thread/connection pool be?" (senior model answer: Little's Law — L = λ·W. At 2000 req/s holding a worker 50 ms, you need ≥100 in flight, so size ≥ 100 plus headroom; set DB pools deliberately below the DB's safe concurrency so backpressure queues in the app, not the database.)
- "Our P50 is great but P99 spiked under load — what's happening?" (senior model answer: a queueing problem, not a work problem. Response time scales as 1/(1−ρ); as utilisation crept toward 1 the tail went vertical. Check utilisation/queue depth, GC pauses, a hot shard or slow replica. Fix by adding capacity to drop ρ, not by optimising the median path.)
- "Why can't we just run servers at 95% CPU to save money?" (senior model answer: at ρ=0.95 mean response is 20× the idle service time, and variance makes it worse; a small spike pushes ρ past 1 and latency diverges with an unbounded queue. Real (M/G/1) variance makes the cliff start earlier. Run hot paths at ρ≈0.5–0.7 and keep headroom for bursts and failover.)
- "Batching doubled our throughput but users complain it's slower — explain." (senior model answer: batching trades latency for throughput. The expensive resource stays busy and amortises fixed cost across the batch, raising req/s, but each request waits to fill/flush the batch, so per-request latency and its variance rise. Tune max-batch-size and a max-wait timeout to bound the latency cost.)
- "Your load test shows clean latency at 2× capacity — do you trust it?" (senior model answer: no — that's the signature of coordinated omission. A closed-loop generator blocks when the server stalls and stops issuing the requests that would have queued, undersampling the slow window. Use open-loop, fixed-schedule load with intended-send-to-completion timing, e.g. wrk2/HdrHistogram.)
- "Define latency and throughput and give a case where improving one hurts the other." (senior model answer: latency = time for one request; throughput = completions/sec. Batching/pipelining raise throughput while raising per-request latency; conversely shrinking batch size or adding hedges lowers tail latency at the cost of extra load/efficiency. They're distinct axes — a fast single request says nothing about capacity.)