Fault tolerance & graceful degradation

At scale, component failure is not an exception — it is the steady state. You do not design to prevent failure; you design so that a failure is non-fatal. The system should degrade, never fail.

First principle

Take 10,000 disks at a 2–4% annual failure rate. That is 200–400 disk deaths per year, or roughly one per day just from disks. Now add NICs, PSUs, kernel panics, OOM kills, expired certs, and bad config pushes. Across a real fleet, something is always broken right now. If your design assumes the happy path, your design is already on fire — you just have not gotten the page yet. The job is to make any single failure a non-event.

Availability is a budget, quoted in nines

Availability is a number you can compute and spend, not a vibe:

A = uptime / (uptime + downtime)

We quote it in nines. Each added nine cuts allowed downtime by 10× — and the marginal cost to buy it climbs steeply.

Availability	Downtime / year	Downtime / day	Roughly what it costs
99% (two nines)	3.65 days	14.4 min	One box, reboot and pray
99.9% (three nines)	8.8 hours	1.4 min	Redundant replicas + a load balancer
99.99% (four nines)	52 min	8.6 s	Multi-AZ, automated failover, on-call
99.999% (five nines)	5.3 min	0.86 s	Active-active multi-region, chaos-tested — often 2× the spend

The senior move

Target the nine the product needs, not the maximum you can engineer. A batch analytics pipeline at five nines is money set on fire; a payments authorization path at three nines is a lawsuit. Pick the SLO per dependency, then spend exactly enough redundancy to hit it. "More nines" is never the answer — "the right nines, cheaply" is.

The two formulas that drive every redundancy decision

Series: dependencies multiply uptime (downward)

A request that needs all n dependencies to succeed has availability equal to their product:

A_series = ∏_i=1..n A_i

Chaining is brutal. Five independent dependencies, each a respectable 99.9%:

0.999⁵ ≈ 0.9950 → you just fell from 3 nines to ~2.5 nines

This is the tyranny of dependencies: every hop you add to the critical path silently subtracts availability. The cheapest reliability win is usually removing a hop, not adding redundancy — make a dependency async (lesson 11), cache it (06), or precompute it off the request path.

Parallel: redundancy multiplies downtime (toward zero)

Put k redundant replicas behind a request that succeeds if any one is up. Now the failures multiply:

A_parallel = 1 − ∏_i=1..k (1 − A_i)

The intuition: a redundant group is unavailable only if every copy is down at the same instant. If each copy is down with probability (1 − A_i) and the failures are independent, the chance they are all down together is the product of those probabilities — so the group's availability is just 1 minus that. Two 99% replicas: 1 − 0.01² = 99.99%. One unreliable component became four nines just by doubling it. Redundancy turns "multiply uptime" into "multiply downtime."

The caveat that bites everyone

The parallel formula assumes independent failures. If your k stateless replicas all read one shared database, one config service, or one DNS zone, that shared thing is a hidden series term. No amount of stateless replication removes it — you have N web tiers in front of a single point of failure. Trace the request to its leaves: the real availability is the product of every distinct dependency, redundancy applied only where it is genuinely independent (callbacks to 03 load balancing and 06 replication).

The resilience toolkit

Each tool prevents a specific failure mode. Know what each one buys and what it costs.

Tool	Prevents	Cost / caveat
Timeouts (mandatory)	One slow dependency hanging every thread/connection until the pool is exhausted	Too tight → false failures; set from the p99 you measured (lesson 02), not a guess
Retries + exp. backoff + jitter	Transient blips (a dropped packet, a brief GC pause)	Useless and dangerous without idempotency (lesson 12); needs a budget
Retry budget	The retry storm: retries piling load onto an already-sick service	Cap retries to ~10% of base traffic, fleet-wide
Circuit breaker	Hammering a drowning dependency; caller latency death	Tuning the open/half-open thresholds; flapping if mis-set
Bulkheads	One exhausted resource pool sinking the whole process	Lower utilization — you partition capacity you can't always share
Load shedding	Total collapse under overload	You drop real work on purpose (forward-ref to lesson 15)

Bulkheads, concretely

The name comes from a ship: watertight compartments mean one breached hull section floods alone instead of sinking the whole vessel. In a service, give each downstream dependency its own connection-pool and thread budget rather than one shared pool for everything. Then if a single slow dependency saturates its pool, the threads waiting on it are confined to that compartment — the threads serving every other dependency are untouched, so one sick downstream can no longer starve the requests that have nothing to do with it.

Retries vs. circuit breakers — opposite reflexes

This is the pairing people get wrong. A retry says "try again, it was probably transient." A circuit breaker says "stop calling, you are clearly down." They are opposite reactions, and the breaker exists precisely because naive retries cause meltdowns.

The retry storm (a positive-feedback meltdown)

A dependency gets slow. Every client times out and retries — on a fixed interval, so they all retry on the same tick. Offered load jumps 2–4×. The sick service, now serving double traffic, gets sicker, times out more, triggering more retries. This is a thundering herd that amplifies the very failure it is reacting to. Defenses, in order: (1) jitter — randomize backoff so clients de-synchronize; (2) exponential backoff — back off harder each attempt; (3) retry budget — cap aggregate retries; (4) circuit breaker — once failures cross a threshold the breaker opens, calls stop entirely and return a fallback instantly, giving the dependency room to recover. Half-open then sends a trickle of probes to detect recovery before fully closing again.

Graceful degradation: the fallback ladder

A dead dependency should yield a worse-but-alive answer, not an error page. Build a ladder of fallbacks; the circuit breaker drives the drop between rungs.

REQUEST │ ▼ ┌──────────────────────────────────────────────────────┐ │ Tier 0 FULL FEATURE live call to dependency │ ◀── best answer └──────────────────────────────────────────────────────┘ │ timeout / breaker OPEN ▼ ┌──────────────────────────────────────────────────────┐ │ Tier 1 CACHED / STALE last-good value (lesson 06) │ ◀── slightly old └──────────────────────────────────────────────────────┘ │ cache miss / cold ▼ ┌──────────────────────────────────────────────────────┐ │ Tier 2 CHEAP DEFAULT heuristic / popular / generic│ ◀── generic but useful └──────────────────────────────────────────────────────┘ │ even that path is down ▼ ┌──────────────────────────────────────────────────────┐ │ Tier 3 STATIC SAFE hard-coded, always-available │ ◀── never errors └──────────────────────────────────────────────────────┘

"A degraded answer beats an error" for most features. State the exceptions: money movement, auth decisions, medical/safety, and anything where a stale or guessed answer is worse than no answer must fail closed — return an explicit error rather than a plausible-but-wrong fallback. Degradation is a default, not a law.

Disaster recovery & multi-region

A single region is one failure domain. A fiber cut, a regional power event, or one bad config push can take it all down at once — redundancy inside the region does not save you. And note that replicas are not backups: a replica faithfully copies a bad DELETE to every copy the instant it happens, whereas a backup is a point-in-time snapshot you can restore from after the mistake. You need both. Two numbers define your recovery posture:

RTO — Recovery Time Objective

How long to come back

RPO — Recovery Point Objective

How much data you can lose

Your RPO equals your async replication lag (callback to 06): if the cross-region stream is 30 s behind when the region dies, you lose up to 30 s of writes. Tighten lag to tighten RPO — at the cost of throughput and latency.

	Active–passive	Active–active
Standby capacity	Idle (cheap)	Full, serving traffic (2× spend)
RTO	Minutes (promote + repoint)	Seconds (already live)
RPO	Async lag	Async lag (or 0 if sync, slow)
Failover path	Exercised only in disasters → risky, untested	The daily steady state → trusted

Two ways multi-region kills you

Split-brain: under a partition both regions think they are primary and both accept writes → divergent, unmergeable state. The fix is quorum + fencing so only one side can win (callback to 08 consensus). The untested standby: a failover path you have never run will fail in anger — the promote script is stale, the DNS TTL is an hour, the standby was never sized for full load. You do not have failover until chaos engineering has proven it by killing replicas and regions in production on a normal Tuesday.

Worked example — should I add redundancy or remove a hop?

A checkout request fans through 5 services, each 99.9% (0.999). Series availability = 0.999⁵ ≈ 0.99501 → 8.8 h/yr per dep becomes ~43.7 h/yr end-to-end. Two options:
(a) Double every service. Each becomes 1 − 0.001² = 0.999999. Chain = 0.999999⁵ ≈ 0.999995 → ~2.6 min/yr. Huge win, but you doubled the fleet (5× the cost) — and only real if the failures are independent.
(b) Remove the two least-critical hops by making them async/cached. Chain = 0.999³ ≈ 0.99700 → ~26 h/yr. Cheaper than (a), and it also cuts latency.
Senior call: do (b) first (free nines from architecture), then apply (a) only to the dependencies that still miss the SLO. Spend redundancy where the math says you must, not everywhere.

Interactive: availability & retry-amplification composer

Set the per-component availability, the depth of the dependency chain, optionally add 2× redundancy per component, and a retry multiplier to see the storm. Watch how series multiplication eats nines and how naive retries amplify load on a sick fleet.

End-to-end availability & retry storm

Per-component availability A. If redundancy is on, each component becomes comp = 1−(1−A)². The chain of n series dependencies is compⁿ. Nines = −log₁₀(1−A). Downtime/yr = (1−A)·31,536,000 s. The retry multiplier R is the extra load each client throws at a failing dependency — effective load ≈ R×, which is the retry-storm amplification.

A (per-component) n (series deps) retry multiplier R add 2× redundancy per component

End-to-end availability

Nines

Downtime / year

Effective load on sick fleet

—

Interview prompts you should be ready for

Five services each at 99.9% sit on a request's critical path — what is the end-to-end availability, and how do you raise it? (senior answer: 0.999⁵ ≈ 99.5%, ~2.5 nines — series availability is the product. Raise it by removing hops first (async/cache/precompute), then adding redundancy only to deps that still miss SLO; redundancy makes comp = 1−(1−A)² and parallelizes failures.)
Why can two stateless replicas behind a load balancer still be a single point of failure? (senior answer: if both read one shared DB/config/DNS, that shared thing is a hidden series term. Parallel availability assumes independent failures; a shared stateful dependency violates it. Real availability = product of distinct dependencies.)
A dependency gets slow and your whole service falls over — what happened and what controls were missing? (senior answer: no timeout, so threads/connections pile up waiting; clients retry on a fixed interval, synchronizing into a retry storm that doubles offered load. Fix: bounded timeouts from p99, backoff + jitter, a retry budget (~10%), and a circuit breaker to stop calling entirely once failures cross a threshold.)
Contrast retries and circuit breakers — when does each apply? (senior answer: opposite reflexes. Retries handle transient blips and require idempotency. A breaker handles sustained failure: it opens to stop hammering a drowning dependency, returns a fallback instantly, protects caller latency, and half-opens to probe recovery. Retry into a healthy-ish dep; break when it is clearly down.)
Describe a graceful-degradation ladder and name a case where you must NOT degrade. (senior answer: full → cached/stale → cheap default → static safe; breaker drives the drops. Do not degrade money movement, auth, or safety-critical paths — a plausible wrong answer is worse than an error, so fail closed.)
Define RTO and RPO, and tell me what sets your RPO. (senior answer: RTO = time to recover, RPO = data you can lose. RPO equals your async cross-region replication lag — 30 s of lag means up to 30 s of lost writes. Tighten lag to tighten RPO, trading throughput/latency.)
Active-passive vs active-active — pick one and defend it. (senior answer: active-passive is cheaper (idle standby) with minutes of RTO but a failover path exercised only in disasters = risky and often broken. Active-active runs failover as the daily steady state, seconds of RTO, full capacity per region, ~2× cost. Choose by whether you can afford untested failover.)
How do you know your failover actually works? (senior answer: you don't until chaos engineering proves it — deliberately kill replicas/regions in production. An untested standby fails in anger: stale promote scripts, long DNS TTLs, under-sized standby. Also guard split-brain with quorum + fencing.)

Takeaway

Failure is the steady state, so design for non-fatal failure, not zero failure. Series dependencies multiply uptime down (remove hops); independent redundancy multiplies downtime toward zero (but watch the shared SPOF). Make timeouts mandatory, retry with backoff + jitter + a budget, break the circuit when a dependency is drowning, and degrade down a fallback ladder so a worse-but-alive answer beats an error — except where you must fail closed. Then prove your multi-region failover with chaos, because an untested standby is no standby at all.