High availability & disaster recovery

A recommender that is down serves zero recommendations — which is strictly worse than one that is slightly stale or slightly worse. So the senior design instinct is not "make it perfect," it is "make it degrade, never fail." Availability is a product feature with a number on it, a budget, and a price that climbs steeply for every extra nine.

Availability is a number, and every nine costs money

Lesson 18 got a good model onto the serving path and defended its latency, rollout, and monitoring. This lesson is the reliability layer wrapped around that path: what happens when a dependency dies, a region goes dark, or the feature store hiccups at 2 a.m. The first move is to stop hand-waving "highly available" and write down the actual target, because the entire architecture follows from which nine you are buying.

Availability is uptime fraction A = uptime / (uptime + downtime), and we quote it in nines. The arithmetic is unforgiving — each nine cuts allowed downtime by 10×:

The cost curve is the whole point. Going from two to three nines is "add a replica." Going from four to five is "run a second region active-active and prove the failover works under chaos" — often a doubling of infra spend and a step-change in operational discipline for ~47 minutes of recovered downtime a year. Chasing an extra nine has steeply rising marginal cost, so a senior engineer asks "what nine does the product actually need?" before architecting. A checkout service may need five; a "people you may know" rail may be fine at three, because its degraded mode (show nobody) is harmless.

SLA vs SLO vs error budget

Three words that get muddled, and the distinction is the spine of an SRE practice:

• SLA — Service Level Agreement. The contractual promise to a customer, with penalties. "99.9% or we credit your bill." External, legal, conservative.
• SLO — Service Level Objective. The internal target you actually engineer to, set tighter than the SLA so you have margin. "We aim for 99.95% so the 99.9% SLA is never at risk."
• SLI — Service Level Indicator. The measured number — the actual success rate / latency you observe.
• Error budget = 1 − SLO. The brilliant reframe: a 99.9% SLO grants you 0.1% of failure — 43 minutes a month — to spend. Spend it on risky launches and faster iteration. Burn it too fast and you freeze launches until it refills. The budget converts "be reliable" (un-actionable) into a quantity engineering and product trade against each other.

The math that drives every architecture decision

Why does redundancy help and why does a long dependency chain hurt? Two formulas, and they govern everything below.

Series (dependencies in a chain). If a request must traverse components 1…n and every one must succeed, availability multiplies:

A_series = ∏_i=1..n A_i

This is the tyranny of dependencies. Five components each at a respectable 99.9% give an end-to-end 0.999⁵ ≈ 0.995 — you fell to two-and-a-half nines just by chaining them. Add a sixth and you drop further. The recsys serving path — gateway → candidate generation → feature store → ranker → re-ranker — is exactly such a chain, which is why every hop you can remove from the critical path buys availability directly.

Parallel (redundant replicas). If you run k interchangeable replicas and the request succeeds if any one is up, you multiply the failure probabilities instead:

A_parallel = 1 − ∏_i=1..k (1 − A_i)

Two replicas at 99% give 1 − 0.01² = 99.99% — two nines became four by adding one box (assuming independent failures, the catch we return to). This is the entire mathematical case for redundancy: parallelism turns multiplication-of-uptime into multiplication-of-downtime, and downtime is the small number.

Redundancy & load balancing — the parallelism, built right

The parallel formula assumes you can actually route around a dead replica. That machinery is stateless replicas behind a load balancer. Stateless is load-bearing: if any replica can serve any request (all state lives in the feature store / cache, not on the box), then the LB can freely spread load and silently drop a sick replica. The instant a replica holds session state, you have lost free failover — that connection's user is stranded.

Why consistent hashing matters for a recommender specifically. The billion-row embedding table of lesson 19 cannot fit on one box — it is sharded. If the routing layer hashes a user/item ID consistently to the same shard, that shard's in-RAM cache of hot embeddings stays warm and the node-local hit rate stays high. With naive round-robin every replica eventually caches everything (memory blowup) or thrashes (cache misses). Worse, when you add or remove a shard, a naive hash(key) mod k remaps almost every key — a cache-cold thundering herd against the backing store. Consistent hashing remaps only the fraction ≈ 1/k that lived on the changed node, so a scale-up is a ripple, not a stampede.

Health checks and N+1 / N+2 capacity

The LB only routes around a dead replica if it knows the replica is dead. Health checks are the mechanism: a shallow check (is the port open?) plus a deep check (can it actually score a synthetic request within budget, with the model loaded and caches warm?). The deep check is what catches a replica that is "up" but serving garbage — the OOM-zombie box that accepts connections and then times out.

Capacity planning is stated as N+1 or N+2: provision enough replicas that you can lose 1 (or 2) and still serve peak traffic within SLA. Run exactly N and the first failure overloads the survivors — who then trip their health checks and get pulled, cascading the whole fleet down. N+2 also covers "one box is down for a deploy when a second one dies."

Multi-region & disaster recovery

A single region is a single failure domain: one fiber cut, one bad config push, one regional power event, and your three-nines-within-the-region service is at zero. Disaster recovery is the layer above redundancy — surviving the loss of an entire region. Two objectives quantify it, and you must state both:

• RTO — Recovery Time Objective. How long until service is restored after a disaster. "Back in 60 seconds."
• RPO — Recovery Point Objective. How much data you can afford to lose, measured in time. "At most 5 seconds of writes." RPO = 0 means zero data loss, which demands synchronous replication and its latency cost.

Why a global recsys runs active-active. Three reasons stack. (1) Latency: serve users from their nearest region — the speed-of-light floor on a cross-ocean round-trip alone can blow the budget. (2) Failover confidence: in active-passive the standby path runs only during a disaster — the worst possible moment to discover it doesn't work; active-active exercises every path continuously. (3) The read-heavy embedding table replicates cleanly: embeddings are written by training and read at serving, so each region keeps a read replica of the embedding table (lesson 19) and the inference path is region-local with no cross-region hop. Writes (logging interactions for lesson 17's streaming updates) are funneled or replicated asynchronously, where small lag is tolerable.

You don't have HA until you've tested the failover

The single most common HA lie is the untested standby. A failover path that has never run in anger will fail in anger — a stale credential, a missing capacity reservation, a DNS TTL nobody lowered. The discipline is failover drills and chaos engineering (Netflix's Chaos Monkey lineage): deliberately kill replicas, regions, and dependencies in production, on purpose, during business hours with everyone watching, and verify the system degrades as designed. GameDays turn the failover from a theoretical capability into a measured, trusted one. If you haven't killed a region on purpose, your RTO is a guess.

Graceful degradation — the recsys-specific superpower

Here is the idea that separates a recommender from a generic web service. A recommender has the rare luxury that a worse answer is almost as good as the right one — a slightly-less-personalized feed still works, where a wrong bank balance does not. So instead of failing when the full model is unavailable, you fall back down a ladder, each rung cheaper, more robust, and less personalized than the one above:

Notice the bottom rung is the popularity prior from lesson 15 — the same maximum-likelihood-under-no-evidence bet that handles a cold user is also your degradation answer for a sick system. That is not a coincidence: "I know nothing about this user" (cold start) and "I can't reach the thing that knows" (degradation) are the same epistemic state, so they share the same fallback. Build the popularity tier once, get both for free.

Circuit breakers — stop hammering the sick service

The mechanism that drives the ladder is the circuit breaker, borrowed from electrical engineering. Wrap each risky dependency in a breaker that watches its error/timeout rate. Three states:

• Closed (normal): requests flow through. Count failures.
• Open (tripped): once the failure rate crosses a threshold, the breaker opens — it stops calling the dependency entirely and immediately returns the fallback (drops to the next tier). This is the key insight: when a dependency is drowning, the kindest thing you can do is stop sending it traffic so it can recover, instead of piling on retries (the retry storm above). The caller also gets a fast fallback instead of waiting for a timeout — so the breaker protects latency too.
• Half-open (probing): after a cooldown, let a trickle of requests through to test if the dependency recovered. Success → close; failure → re-open. This auto-heals without a human.

Circuit breakers travel with three companions. Timeouts cap how long any single call may block (no timeout = one slow dependency hangs every thread). Bulkheads isolate resource pools per dependency (like a ship's watertight compartments) so a flood in one — say the ranker's thread pool exhausting — can't sink the gateway that also serves the popularity fallback. Load shedding is the deliberate drop of the lowest-value requests when overloaded — shed the exploration traffic and the logging fan-out before you shed a paying user's core request, keeping the service alive within capacity rather than collapsing under it.

Interactive · availability & degradation calculator

Set each component's availability and how many sit in series on the critical path. Then watch two senior moves rescue it: adding a redundant parallel replica (each component becomes two, by the 1−∏(1−A) rule) and adding a graceful-degradation fallback tier that catches requests the primary path drops. The series multiplication is the villain; redundancy and degradation are the two heroes.

Stateful hazards — where the parallel formula lies

The clean 1−∏(1−A) math assumes independent failures. The dangerous reality is the shared, stateful component that all your "redundant" replicas depend on — a hidden series term that no amount of stateless replication removes. These are the real single points of failure in a recsys:

• The feature store as a SPOF. Every stateless ranker replica reads from it; if it's down, all your redundancy is moot — you've parallelized the cheap part and left the expensive shared dependency single. Fix: replicate the store itself, and put a circuit breaker in front so a slow store drops you to cached/popularity tiers instead of hanging every replica.
• The model registry / config service. A box that fetches its model from a central registry on startup will fail to come up during a deploy if the registry is down — and that's exactly when you're restarting boxes. Bake a last-known-good model into the image so a replica can boot without the registry.
• Cache stampede on expiry. A hot key expires; the next thousand concurrent requests all miss simultaneously and all hammer the backing store to recompute it — a self-inflicted thundering herd. Fixes: request coalescing / single-flight (one request recomputes, the rest wait on it), staggered TTLs with jitter (don't expire a whole class of keys on the same tick), and serve-stale-while-revalidate (return the expired value and refresh in the background — degradation in miniature).
• The model that OOMs and takes the box down. A pathological request (an enormous candidate set, a memory leak under a new model) blows the box's memory; the OS OOM-killer reaps the process, the box fails its health check, traffic shifts to the survivors — who now face the same pathological pattern and OOM in turn. This ties straight to lesson 18's monitoring: memory and fallback-trigger-rate are the upstream signals that move before the cascade; cap candidate-set size, set memory limits, and let load shedding drop the monster request rather than letting it kill the host.

Interview prompts you should be ready for