Horizontal scaling & load balancing
One box is at 80% utilisation and the graph is still climbing. Your next move decides whether the system can ever survive a single dead machine. This lesson is about making that move on purpose.
1. Scale up vs scale out
When demand outgrows a single server you have exactly two levers. Vertical scaling (scale up) means a bigger box: more cores, more RAM, faster disk. Horizontal scaling (scale out) means more boxes behind a distributor. They are not equivalent, and the difference is the whole lesson.
| Property | Vertical (scale up) | Horizontal (scale out) |
|---|---|---|
| Code change | None — same process, same assumptions | Tier must become stateless + a coordination layer |
| Cost curve | Super-linear: top-bin CPUs/RAM carry a premium | ~Linear: commodity boxes, buy N of them |
| Ceiling | Hard — largest instance that exists | Soft — keep adding nodes (until a shared dependency caps you) |
| Failure domain | One. Box dies → service dies | N. Lose a node, the rest serve on |
| When it shines | Stateful single-writer DBs, quick wins, low ops | Stateless app/web tier, large fleets, HA requirements |
Vertical is genuinely the right first move sometimes: it is free engineering-wise and keeps you on one consistent machine (no distributed-systems tax). But it buys time, not safety — a single 96-core box at 50% utilisation is still a single failure domain. The moment availability matters, you are scaling out, because only horizontal scaling adds redundancy. Adding the second box is never about throughput first; it is about surviving the loss of the first box.
2. Stateless is the magic word
Horizontal scaling only delivers free scaling and free failover if any replica can serve any request. That property has a name: statelessness. A replica is stateless when no per-client state lives on the box — the request carries everything needed, or the state is fetched from a shared store.
Where does the session go, then? Three standard homes:
- Shared session store (Redis/Memcached/DB): every replica reads the same place. Sets up lesson 06 (caching) and 06 (replication).
- Signed token / cookie (JWT, signed session cookie): the client carries its own state; the server verifies a signature and holds nothing. Zero server-side lookup, at the cost of revocation difficulty and token size.
- Distributed cache for derived state: reconstruct on miss.
The instant a replica holds the only copy of a user's session, two things break at once. Failover breaks: that replica dies and the user is stranded (logged out, cart lost). Scaling breaks: the LB can no longer route freely — it must pin that user to that box (a sticky session), so load can't rebalance and a hot user can't be moved off a hot node.
Statelessness has a quieter partner that the LB forces you to confront: idempotency. A load balancer (or the client) that times out and retries a request can deliver the same write twice — the first attempt may have succeeded and only the response was lost. A non-idempotent write then double-executes (the card is charged twice, the order placed twice). An idempotent endpoint — one where applying the same request twice has the same effect as applying it once — makes that retry safe. This is why "stateless + idempotent" is the combination that lets you retry freely: stateless means any replica can take the retry, and idempotent means it doesn't matter if the original already ran. See 03 for how idempotency keys make a write safely repeatable, and 12 for why idempotency is the workhorse that turns unreliable, at-least-once delivery into a correct result.
3. The load balancer
Once there are N replicas, something must spread requests across them. That distributor is on the critical path of every request, so it is both essential and a prime single point of failure. You run it redundantly (active-active pair, or a managed LB that is itself a fleet) — an LB that can take the whole site down is worse than no LB.
L4 vs L7
| L4 (transport) | L7 (application) | |
|---|---|---|
| Routes by | IP + port (TCP/UDP) | HTTP path, header, cookie, method |
| Sees payload? | No — protocol-opaque | Yes — parses HTTP, can terminate TLS |
| Cost / throughput | Cheap, very high pps, low latency | More CPU per request (parsing, TLS) |
| Can do | Fast fan-out, NLB-style | Path routing, canary by header, WAF, retries, gzip |
| SPOF weight | Simple, hard to break | Richer, more logic to go wrong |
Globally, before any of that, you have coarse distribution: DNS round-robin hands out rotating A-records (simple, but clients/resolvers cache and TTLs lag failures), and anycast announces one IP from many sites so BGP routes each client to the nearest PoP. These spread load between regions/PoPs; L4/L7 LBs spread it within a region. Most real systems use all of them stacked.
Note the granularity ladder, coarse to fine: anycast/DNS picks a region → L4 LB picks a fleet → L7 LB picks a replica by request content. Each layer is independently redundant; a failure at one layer should never hard-depend on a single instance at the next.
LB algorithms
| Algorithm | Idea | Trade-off |
|---|---|---|
| Round-robin | Next replica in rotation | Dead simple; ignores actual load & request cost |
| Weighted RR | RR with per-node weights | Handles heterogeneous boxes; weights are static guesses |
| Least-connections | Send to fewest open conns | Good when request cost varies; conns ≠ load for keep-alive |
| Least-response-time | Fewest conns + lowest latency | Adapts to slow nodes; needs live measurement |
| Consistent hashing | hash(key) → same node | Affinity for cache locality (lesson 07); uneven without vnodes |
| Power-of-two-choices | Pick 2 at random, take the less loaded | Near-optimal balance with O(1) state; avoids the herd |
4. Health checks & capacity planning
The LB must stop sending traffic to broken nodes. Two depths:
- Shallow check — is the port open / does
/healthzreturn 200? Cheap, frequent. Catches a dead process; misses a process that's up but useless. - Deep check — can it actually serve a synthetic request (touch the DB/cache, do real work) within a latency budget? Catches the OOM-zombie, the node with a wedged connection pool, the "200 but 8-second" box. More expensive, run less often, and beware: a deep check that hits a shared dependency can take the whole fleet out of rotation if that dependency hiccups.
N+1 / N+2 and cascading collapse
The point of running multiple replicas is to survive losing some. N+k provisioning means you carry k spare replicas beyond the bare minimum N needed to serve peak at safe utilisation, so you can lose k nodes and still meet SLA. Run exactly N (k = 0) and the first failure is fatal in a specific, vicious way:
Autoscaling adds and removes nodes automatically. Reactive (scale on CPU/QPS thresholds) is simple but lags — by the time the metric crosses, you're already behind, and cold-start/warm-up (JIT, connection pools, cache fill) means new capacity isn't useful for tens of seconds. Predictive/scheduled scaling pre-warms ahead of known peaks. Always scale in conservatively: yanking capacity right before a spike re-creates the exact zero-headroom cascade above.
A subtle trap: autoscaling on CPU assumes CPU is your bottleneck. If a downstream DB or connection-pool limit is the real cap, adding app replicas just multiplies the pressure on the shared dependency and accelerates its collapse. Always scale on the metric that reflects the actual constraint, and verify the next tier can absorb the fan-out before you let the fleet grow.
Bare minimum N = ⌈12000 / 2000⌉ = 6 replicas (at 100% of safe capacity — zero headroom, the cascade waits).
After losing 1 of 6: per-node load = 12000 / 5 = 2,400 req/s > 2,000 → overload. So 6 isn't even N+1.
For N+2 (survive 2 failures): need 12000 / (M − 2) ≤ 2000 → M − 2 ≥ 6 → M = 8 replicas.
Check: lose 2 of 8 → 12000 / 6 = 2,000 = safe cap exactly. Fleet utilisation at full strength = 12000 / (8 × 2000) = 75% of safe capacity. That 25% slack is the price of surviving two dead boxes — and it is cheap insurance.
5. Capacity & failure-headroom calculator
Set your peak load and per-replica safe capacity, then see how many nodes you can actually afford to lose. The "+k you really have" is the number that matters in the postmortem.
Interview prompts you should be ready for
- "Your service is at 80% CPU on one box. Walk me through your next move." (senior answer: confirm it's the app tier not the DB; if availability matters, scale out not up, because only adding boxes adds a second failure domain — a bigger box is still one box. First make the tier stateless: push session to a shared store or signed token. Then put a redundant LB in front. Size for N+2 against peak at safe ρ.)
- "What does 'stateless' actually buy you, concretely?" (senior answer: any replica serves any request, so the LB routes freely and silently drops a dead replica with zero user impact — free failover and free scaling. The cost is an extra network hop to the shared state store and the need to make that store HA, which is what lessons 06–08 are about.)
- "L4 or L7 load balancer here — why?" (senior answer: L7 if I need path/header routing, TLS termination, canary, or a WAF — it sees HTTP. L4 if I just need raw fan-out at very high packet rates with minimal latency and the routing logic is trivial. Many stacks layer L4 in front of L7. State the CPU cost of L7 parsing as the trade.)
- "Round-robin is balancing poorly — requests have wildly different costs. What do you switch to?" (senior answer: least-connections or least-response-time so expensive in-flight requests steer new ones away; or power-of-two-choices for near-optimal balance with O(1) state. Avoid exact 'least-loaded' — every LB stampedes the same node before metrics refresh.)
- "Why not just enable sticky sessions and move on?" (senior answer: stickiness pins a user to a box, so you lose free failover — that box dies and every pinned user is stranded — and load can't rebalance. It's masking missing externalised state. Acceptable only as a deliberate exception, e.g. long-lived WebSockets, never as the default fix.)
- "You run exactly enough replicas for peak and one dies. What happens?" (senior answer: its traffic redistributes, survivors cross safe ρ, latency spikes per the utilisation law, deep health checks time out, the LB pulls them, and the fleet cascades to total collapse within seconds. That's why I provision N+1/N+2 and keep fleet utilisation at ~70–80% of safe capacity.)
- "Shallow vs deep health checks — which and how often?" (senior answer: shallow/liveness frequently to catch dead processes cheaply; deep/readiness less often to catch up-but-broken nodes — wedged pools, OOM zombies, 200-but-slow. Guard the deep check so a shared-dependency hiccup can't pull the entire fleet out of rotation at once.)
- "Autoscaling didn't save us during a traffic spike. Why?" (senior answer: reactive scaling lags — the threshold trips after you're already behind, and new nodes have cold-start/warm-up before they're useful, so you eat a few minutes of degradation. Fix with predictive/scheduled pre-warming for known peaks and conservative scale-in so you never strip headroom right before a surge.)