Optimisation playbook
The earlier lessons each dropped an optimisation as an aside — cache the read path, batch the calls, pick the right index, shed the fan-out. This lesson gathers them into one menu and, more importantly, teaches the discipline that decides which one to reach for. The headline: optimisation is not "make code fast," it is "find the dominant term and make it smaller — then re-measure." Most engineers lose weeks optimising the wrong term.
A system's total cost (latency, money, load) is a sum of terms. Speeding up any term yields a return proportional to how big that term already is. So the entire game reduces to: (1) measure to find the biggest term, (2) attack it with the cheapest lever that shrinks it, (3) re-measure, because the bottleneck has now moved. The cheapest win is almost always doing less work, not doing the same work faster.
The discipline: measure, attack the dominant term, repeat
Before any technique, the method. Optimisation done by intuition is gambling — the part of the code you feel is slow is routinely not the part the clock agrees is slow. So the cardinal rule is a loop, not a step:
Three corollaries you should be able to recite:
- Measure before you optimise. You cannot optimise what you have not measured; profiling and tracing (16) tell you which term dominates. A guess that happens to be right still teaches you a bad habit.
- Optimise the dominant term. Effort spent on a small term is wasted no matter how clever it is — the next section (Amdahl) makes this precise.
- The cheapest win is doing less work. Not computing a result, not making the call, not moving the bytes — these beat any amount of making the work faster. A cache hit is the limit case: the fastest query is the one you never run.
Amdahl's law — why the dominant term is the only one worth chasing
Start with numbers, not algebra. A request takes 100 ms: 60 ms in the database, 40 ms everything else. Suppose you make that "everything else" infinitely fast — zero. You are left with 60 ms. You worked miracles on 40% of the budget and the request is still 60 ms: a 1.67× speedup, and that is the best you could ever do by touching only that slice. Conversely a mere 2× on the 60 ms DB slice (→ 30 ms) gives 100/70 = 1.43× — already comparable, from far less heroics.
Now the formula. If a fraction p of the time can be sped up by a factor s, the overall speedup is:
speedup = 1 / ((1 − p) + p/s)
Push s → ∞ (make that slice free) and the p/s term vanishes, leaving the hard ceiling:
max speedup = 1 / (1 − p)
That ceiling is the whole lesson in one line: you cannot speed the system up by more than the share of time you are actually attacking allows, however brilliant the optimisation. Optimising a 5% slice caps you at 1/(1−0.05) ≈ 1.05× — a rounding error. The intuition: chase the big slice.
| Fraction p you optimise | Ceiling 1/(1−p) (even at s=∞) | Speedup at a realistic s=4× | Reading |
|---|---|---|---|
| 0.05 (a 5% slice) | 1.05× | 1.04× | not worth touching |
| 0.20 | 1.25× | 1.19× | marginal |
| 0.50 | 2.0× | 1.60× | worth it |
| 0.80 | 5.0× | 2.5× | the bottleneck — attack here |
| 0.95 | 20× | 3.48× | huge leverage |
Amdahl is also why the loop never ends: once you halve the 80% slice it becomes ~67% of a smaller total, and some other term is now the dominant one. The bottleneck moves; you re-measure and chase the new biggest slice.
The optimisation ladder — cheapest, highest-leverage first
Climb this top-down. Each rung is harder and riskier than the one above, so exhaust the cheap rungs before reaching for concurrency or more machines. For every rung: what it buys, what it costs.
Rung 1 — Do less work (the highest-leverage rung)
The work you skip is free and never fails. This is where the biggest, cheapest wins live.
- Cache hot reads (06): a hit serves a result you'd otherwise recompute or re-fetch. Buys: latency and backend load collapse. Costs: a second copy that can go stale, plus a new failure tier.
- Precompute / materialise (19): compute the answer at write time (a materialised view, a denormalised counter) so reads are a lookup. Buys: read latency. Costs: write amplification and staleness.
- Avoid the N+1 (03): one query that fetches N rows beats N queries fetching one row each. Buys: N−1 round trips removed. Costs: a slightly fatter query to write.
- Cap the work per request: paginate, set
LIMIT, reject unbounded queries. Buys: bounded worst case. Costs: clients must page.
Rung 2 — Move work off the hot path
If work must happen but the user needn't wait for it, get it off the critical path. This shrinks p for the synchronous request even though total work is unchanged.
- Async queues / background jobs (11): reply now, do the slow work later (emails, thumbnails, indexing). Buys: tail latency off the request path; burst absorption. Costs: eventual consistency, a queue to operate, harder debugging.
- Batching & pipelining (02): amortise fixed per-call cost over many items. Buys: throughput / efficiency. Costs: per-request latency and variance rise as items wait for the batch.
Rung 3 — Reduce round trips & bytes moved
The latency ladder (01) is the reason this rung matters: one network round trip (~0.5–150 ms) dwarfs a thousand memory reads (~100 ns each). A request that makes ten sequential hops has already lost the game before any code runs.
- Collapse hops: fewer sequential network calls; co-locate chatty services. Buys: each removed RTT is pure latency saved. Costs: coupling, fatter services.
- Batch endpoints (03): one request returning many resources instead of many requests. Costs: a coarser, less cacheable API.
- Projection — read only the columns/fields you need:
SELECT id, namenotSELECT *. Buys: less I/O, smaller payloads, narrower index-only scans. Costs: queries coupled to needed fields. - Compression & binary encodings (gzip/zstd; protobuf/Avro over JSON). Buys: fewer bytes on the wire. Costs: CPU to (de)serialise; harder to eyeball.
Rung 4 — Make each unit of work faster
Only now do we make the work itself faster — the rung most people start on, and usually too early.
- The right index (04): turns an O(N) table scan into an O(log N) seek. The single highest-leverage move inside a database. Costs: slower writes and storage per index.
- Better algorithm / data structure: an O(N²) inner loop is a bottleneck waiting to happen; a hash lookup beats a linear scan. Costs: code complexity, sometimes memory.
- Connection pooling & keep-alive: reuse TCP/TLS connections instead of paying handshake setup (~1–2 RTT) on every call. Buys: per-call setup amortised away. Costs: pool sizing (size it below the DB's safe concurrency — see Little's Law, 02).
Rung 5 — Parallelise / scale out
When the work is irreducible and already lean, do it in parallel or on more machines. Powerful, but the most expensive rung and the one with a sharp tail-latency trap.
- Concurrency & fan-out: split a request across workers/shards and combine. Buys: wall-clock latency for parallelisable work; throughput. Costs: tail-at-scale (02) — waiting on the max of n backends means more fan-out makes p99 worse, not better. Mitigate with hedged requests and shallower fan-out.
- Horizontal scale (05): add replicas/shards to lower utilisation per node (drops the 1/(1−ρ) wait, 02). Costs: money, coordination, partitioning (07).
Rung 6 — Right storage tier
Match where data lives to how hot it is, using the latency ladder (01): RAM ≫ SSD ≫ spinning disk ≫ network. Keep the working set (the data actually touched) in RAM; let cold data fall to cheaper tiers. Buys: orders-of-magnitude latency on hot data. Costs: RAM is expensive; tiering adds complexity.
The techniques, consolidated
One table to scan in an interview. Note every row names a cost — a technique without a stated cost is a technique you don't yet understand.
| Technique | What it buys | What it costs / when it backfires |
|---|---|---|
| Caching (06) | Latency + backend load collapse on hot reads | Staleness, invalidation, a new failure tier; useless for write-heavy or unique reads |
| Precompute / materialise (19) | Reads become cheap lookups | Write amplification; stale views; rebuild cost |
| Avoid N+1 / batch queries (03) | N−1 round trips removed | Fatter queries; over-fetching if careless |
| Async / background jobs (11) | Tail work off the hot path; burst absorption | Eventual consistency; a queue to run; debugging |
| Batching / pipelining (02) | Throughput, hardware efficiency | Per-request latency & variance rise |
| Fewer hops / collapse RTTs (01) | Each removed round trip is pure latency saved | Service coupling, coarser boundaries |
| Projection / compression / binary (03) | Fewer bytes moved & parsed | (De)serialise CPU; less human-readable |
| Right index (04) | Scan → seek, O(N) → O(log N) | Slower writes, storage per index; wrong index unused |
| Better algorithm / data structure | Lower asymptotic cost per call | Complexity; sometimes more memory |
| Connection pooling / keep-alive | TCP/TLS setup amortised away | Pool sizing; exhaustion stalls the app |
| Parallelise / fan-out | Wall-clock latency, throughput | Tail-at-scale: more fan-out → worse p99 |
| Horizontal scale (05) | Lower ρ per node; capacity | $, coordination, partitioning complexity |
| Hotter storage tier (01) | Orders-of-magnitude on hot data | RAM cost; tiering complexity |
The traps
- Premature optimisation. Optimising before measuring, or before the code matters, trades real complexity for an imaginary gain. Make it correct, measure, then optimise the term that proves dominant.
- Optimising a non-bottleneck (Amdahl). Sinking effort into a 5% slice caps your reward at ~1.05× no matter how clever. Always ask "what fraction of the budget is this?" before touching it.
- Micro-tuning code while a network hop dominates. Hand-optimising a function that runs in microseconds while a 50 ms RTT sits next to it in the trace. One round trip dwarfs a thousand memory reads (01) — fix the hop first.
- A cache that buys speed with staleness (06). If the data must be fresh, a cache trades a correctness bug for a latency win. Wrong tool when the read is unique, write-heavy, or strongly-consistent.
- Over-parallel fan-out that worsens the tail (02). Splitting a request across more backends to "go faster" makes you wait on the slowest of more leaves — p99 climbs. Past a point, fan-out is a tail-latency generator, not an optimisation.
Try it: an Amdahl latency-budget optimiser
A request's 100 ms budget is split across four components. Pick one to optimise, set a speedup factor, and watch the overall speedup — and the Amdahl ceiling for that slice (the best you could ever do at infinite speedup). The lesson lands when you optimise the 5 ms serialisation slice 20× and barely move the total, then put a modest 2× on the DB slice and save real milliseconds.
Interview prompts you should be ready for
- "This endpoint is slow. Where do you start optimising?" (senior answer: I don't guess — I measure first. Profile or trace the request (16) to find the dominant term, then attack that term with the cheapest lever that shrinks it, then re-measure because the bottleneck moves. The fastest win is usually doing less work, not faster work.)
- "How do you optimise a slow endpoint without guessing?" (senior answer: break the latency budget into terms — DB, network, compute, serialisation — via a trace; the largest term is the target. Amdahl says effort on a small term is wasted, so I confirm the slice is big before touching it, change one thing, and re-measure. Guessing is how teams burn weeks on a 5% slice.)
- "What's the ceiling on optimising component X if it's only 20% of latency?" (senior answer: 1/(1−p) = 1/(1−0.2) = 1.25× — even if I make X infinitely fast. So at best the request gets 25% faster; if I need more, I must attack a bigger slice. That ceiling is exactly why I measure shares before optimising.)
- "When is caching the wrong tool?" (senior answer: when reads aren't repetitive (low hit rate, unique keys — you pay for a cache that never hits), when the workload is write-heavy, or when the data must be strongly fresh — then a cache trades a correctness bug for a latency win. Cache buys speed with staleness; if staleness isn't affordable, reach elsewhere (06).)
- "We added more parallel fan-out and p99 got worse — why?" (senior answer: tail-at-scale (02). A request waiting on all n backends sees the max of n latencies, so more fan-out means a higher chance some leaf is slow; p99 climbs with n. Fixes: shallower fan-out, hedged/backup requests so the max collapses toward the median, timeouts with partial results.)
- "Engineer A wants to rewrite a hot function in C; the trace shows a 60 ms DB query and 2 ms in that function. What do you say?" (senior answer: that's micro-tuning a non-bottleneck. The function is ~2% of the budget — ceiling ~1.02×. The DB query is 60%, ceiling 2.5×; a single right index or removing an N+1 there saves far more than any rewrite. Attack the dominant term.)
- "Give the optimisation ladder from cheapest to most expensive." (senior answer: do less work (cache, precompute, kill N+1, cap pages) → move work off the hot path (async, batch) → reduce round trips and bytes (collapse hops, projection, compression) → make each unit faster (index, algorithm, pooling) → parallelise/scale out (mind the tail) → right storage tier (working set in RAM). Exhaust the top rungs first.)
- "You optimised the bottleneck and it's still slow — now what?" (senior answer: re-measure. Halving the dominant term shrinks the total and promotes some other term to dominant — the bottleneck moved. Optimisation is a loop: measure, attack, re-measure, until the budget meets the SLO or the next term costs more to fix than it's worth.)
Optimisation is a discipline, not a bag of tricks: measure, attack the dominant term, re-measure. Amdahl gives the organising law — your reward is capped at 1/(1−p), the share of time you actually attack, so chase the big slice and never the 5% one. Climb the ladder cheapest-first: do less work, then move work off the hot path, then cut round trips and bytes, then speed each unit, and only then parallelise or scale out — every rung naming its cost. The traps are all the same mistake in disguise: optimising something that isn't the bottleneck. Next, 15 · Rate limiting & the API edge — where the cheapest optimisation of all, refusing work you shouldn't do, lives at the front door.