Part III - Production inference

Autoscaling and AI-Aware Routing

Lesson 08 taught you to measure a single replica — to find the batch size and concurrency where one GPU delivers the most tokens per second without blowing past your TTFT budget. But production runs many replicas behind a load balancer, and two new questions appear at once: when do you add a replica, and which replica should each request go to? A naive Kubernetes setup answers both badly — it scales on CPU and balances at random — and for LLM serving both defaults are actively harmful. This lesson replaces them with signals and routing that match how LLM inference actually consumes a GPU: tokens, queue time, cache state, and adapter locality.

1 · The router is a scheduler above the scheduler

Kubernetes already has a scheduler: it places pods onto nodes. But once your model server is running, a second scheduling decision happens on every single request — which running replica handles this prompt? By default a Kubernetes Service answers that question by spraying connections across endpoints roughly at random (round-robin / random L4 balancing). For a stateless web app that is correct: every pod is identical, so any pod is as good as any other.

LLM replicas are not identical, even when they run the same model image. Each replica carries hidden state that makes it better or worse for this specific request:

The mental model: an LLM router is a scheduler above the scheduler. The Kubernetes scheduler decides where pods live; the router decides which warm model, cache state, adapter, and queue should absorb the next request. It is the smallest piece of machinery that turns a flat pool of "identical" replicas into a fleet whose differences you can exploit. The cost is that routing becomes stateful and policy-heavy — fairness, cache locality, and latency can pull in different directions — which is exactly why the rest of this lesson is about choosing the right signal for each decision.

2 · Why a CPU HPA fails — scale on what binds

The default autoscaler is the HorizontalPodAutoscaler (HPA) driving on CPU utilization: when average pod CPU crosses, say, 70%, add a replica. This is the single most common mis-configuration in LLM serving, and it fails in both directions.

The reason is structural. LLM inference is bound by the GPU and by GPU memory bandwidth, not by the host CPU. A model server can have its GPU pinned at 100% — every streaming multiprocessor busy doing matrix multiplies, the KV cache full, requests piling up in the queue — while the host CPU, whose only job is to copy bytes and dispatch CUDA kernels, sits at 15%. The HPA looks at 15% CPU, concludes the pod is idle, and refuses to scale. Meanwhile TTFT is climbing because requests are waiting in the queue. Worse, on a quiet day the CPU can briefly spike on tokenization and trick the HPA into scaling up when the GPU is nearly idle, wasting an expensive accelerator. CPU is uncorrelated with the resource that actually binds.

Scale on the signals that bind LLM serving instead. None is universally best; they trade responsiveness against stability:

In practice a robust policy scales up on queue time or KV-cache pressure (early warning) and uses token throughput to size the steady-state replica count, while never trusting CPU. Custom and external metrics reach the HPA through the Kubernetes custom.metrics.k8s.io API, fed by an adapter such as Prometheus Adapter or KEDA scraping the runtime's /metrics endpoint.

3 · Cache-aware routing — the worked number

Here is where random load balancing does real damage. Recall from lesson 08 that prefill (processing the prompt) is compute-heavy, and that a runtime keeps a prefix cache: if a new prompt shares a leading prefix with one it already processed — a shared system prompt, a few-shot preamble, a conversation history — the replica reuses the cached KV state for that prefix and skips recomputing it. A prefix-cache hit can cut prefill compute and TTFT for that request to near zero.

But the prefix cache is local to each replica. Replica A's cache does not help a request that lands on replica B. So the routing decision directly determines your hit rate. Random balancing scatters requests that could have hit a warm prefix across the whole fleet, where most land on a replica that never saw that prefix — a miss. Cache-aware routing instead sends requests that share a prefix to the same replica, concentrating hits.

The widget below lets you drag the two knobs — number of replicas and the shareable fraction — and watch random vs. cache-aware hit rate diverge. The gap widens as you add replicas, because random's 1/R term shrinks while cache-aware does not depend on R at all.

// shareable fraction s, replica count R, warm-when-aware ~0.80
const awareHit  = s * 0.80;            // shared prefixes pinned to one replica
const randomHit = s * (1 / R);         // shareable only hits if it lands on the warm replica
// prefill on a hit is ~free; if prefill is ~half of GPU compute,
// compute saved ≈ 0.5 * hitRate.

4 · LoRA-aware routing, the AI gateway, and a K8s-native standard

LoRA-aware routing. The same locality logic applies to fine-tunes. A LoRA adapter is a small set of weight deltas (often tens to a few hundred MB) layered on a shared base model, so one server can serve many fine-tunes by swapping adapters. But swapping an adapter that is not resident means fetching and loading it — adding latency on the request's hot path and a burst of memory traffic. LoRA-aware routing keeps a live inventory of which adapters are warm on which replicas and routes a request for adapter X to a replica that already has X loaded. The trade-off mirrors cache-aware routing: you gain locality at the cost of maintaining adapter-residency state and accepting less perfectly even load.

The AI gateway. All of this — auth, rate limits, prompt-logging policy, model selection, traffic shaping during a rollout — wants a single boundary in front of the pools. That boundary is the AI gateway: the place where a request is authenticated, checked against a tenant's quota, tagged with the model/adapter it is asking for, logged according to policy, and then handed to the router. Keep the boundary honest: the gateway must not hide runtime metrics, and the model-server endpoints must remain directly observable during an incident — otherwise the gateway becomes a blindfold exactly when you need to see.

Gateway API Inference Extension. The emerging Kubernetes-native way to express this is the Gateway API Inference Extension, a SIG-Network project that extends the Gateway API for inference workloads. It is maturing rather than long-settled — APIs are still evolving and you should pin versions and read the project's status before betting on it — but the design is the right shape, which is why it is worth knowing. Its central idea is the InferencePool: instead of treating a Service as a flat set of interchangeable pods, the gateway treats an inference pool as a backend that has an endpoint picker (EPP). The picker looks at inference-specific signals — queue length, KV-cache utilization, which LoRA adapters are active on each replica, and model-server readiness — before choosing the backend. That is precisely the cache-aware / LoRA-aware / readiness logic of this lesson, lifted into a shared API rather than re-implemented per team.

5 · Failure modes and the implementation checklist

Checkpoint exercise

Where this points next

Cache-aware and LoRA-aware routing both rest on a hidden assumption: that a single replica does both prefill (compute-bound) and decode (memory-bandwidth-bound), so "warm prefix on this replica" is a meaningful, stable property. But those two phases stress the GPU so differently that the next architectural move is to split them onto separate pools — a prefill tier and a decode tier — and stream the KV cache between them. That changes everything about routing and scaling: you now scale and route each tier on its own bottleneck, and the cache locality you just learned to exploit becomes a thing you ship across the network. Lesson 10, Disaggregated Serving, builds that prefill/decode split and the KV-transfer machinery it requires. (For a deeper look at how a runtime actually implements the cache-aware scheduling this lesson routes toward, see SGLang cache-aware scheduling.)

Interview prompts

• Why does a CPU-based HPA fail for LLM serving, and what do you scale on instead? (§2 — LLM inference is GPU/HBM-bound, not CPU-bound, so CPU reads idle while the GPU queue saturates and TTFT climbs; scale on token throughput, queue time, and GPU/KV saturation, exported via custom/external metrics.)
• Why is random load balancing harmful for LLMs when it is correct for a stateless web app? (§1, §3 — LLM replicas carry hidden per-replica state — prefix cache, loaded adapters, queue/KV pressure — so they are not interchangeable; random spray scatters shareable prefixes and drops the hit rate to ~1/R.)
• Work the cache-aware vs. random hit-rate number for 8 replicas and 70% shareable traffic. (§3 — random ≈ 0.70 × 1/8 ≈ 9%; cache-aware ≈ 0.70 × 80% ≈ 56%; the shareable slice goes from ~12% (the 1/R floor) to ~80% hit rate, cutting prefill compute on that slice proportionally.)
• What is the cold-replica trap and how do you prevent it? (§2 — a scaled-up replica is network-reachable before weights finish loading into HBM, so it serves errors/30 s TTFTs during the overload that triggered it; gate the readiness probe on weight-load completion so the Service/router excludes it.)
• What is LoRA-aware routing and what does it cost? (§4 — keep an inventory of which adapters are warm on which replicas and route adapter X to a replica that already has X, avoiding an on-path load; the cost is maintaining adapter-residency state and accepting less even load.)
• What does the Gateway API Inference Extension add over a normal Service? (§4 — an InferencePool backend with an endpoint picker that routes on inference signals — queue length, KV utilization, active LoRA adapters, readiness — making cache/LoRA/readiness-aware routing a shared (still-maturing SIG) API instead of a per-team hack.)
• Why is the endpoint picker a fail-open/fail-close decision? (§4 — the picker is on the request path, so if it is unavailable you must choose: fail-open routes blindly to protect availability, fail-close rejects to protect correctness/overload boundaries; either way it belongs in observability and incident drills.)