Part I - Inference foundations

Why GenAI Is Different on Kubernetes

This track starts where most teams start: you already know how to ship a stateless web service on Kubernetes — a container listens on a port, replicas sit behind a Service, and a Horizontal Pod Autoscaler (HPA) adds pods when CPU rises. That instinct is the problem. A large language model (LLM) server breaks every assumption that instinct rests on: it carries a heavy artifact that must load before it can answer, its request is not one transaction but a long stream of token steps, and the resource it competes for is not CPU but a few gigabytes of scarce accelerator memory. This lesson separates the model problem from the platform problem, so that everything later in the track — manual deployment, autoscaling, routing, multi-tenancy — has a frame to hang on.

Source coverage

PDF Introduction: the challenges of running generative AI at scale, why Kubernetes is the substrate for AI workloads, and the operational LLM fundamentals (tokenization, prefill/decode, KV cache, model artifact size).

Linear position

Prerequisite: general backend/cloud experience and basic Kubernetes (pods, Deployments, Services, HPA) plus basic ML. This is lesson 1 — there is no prior lesson; we open from the track premise. No GPU/LLM-serving specifics are assumed; every term is defined at first use.
New capability: the ability to look at a GenAI workload and immediately separate the platform problem from the model problem — the model predicts tokens, but the platform must manage artifacts, accelerators, request queues, identity, state, and cost — and to name which scarce resource binds first.

The plan

Five moves. (1) Show why the web-service mental model breaks: an LLM request is artifact-readiness plus prefill plus many decode steps plus streaming plus policy, not one HTTP transaction. (2) Teach the operational LLM fundamentals — tokenization, prefill, decode, KV cache, TTFT, TPOT, HBM — only as deep as the platform needs them. (3) Work a concrete numeric example: a 70B model on an 80 GB GPU, showing exactly why a CPU HPA and a normal readiness probe fail. (4) Reframe the first trade-off — not "which Kubernetes object?" but "why am I running this model myself?" — and give the "four numbers" that route you to a storage, GPU-memory, routing, or cost problem. (5) Failure modes, a checklist, and the hand-off into the first manual deployment.

1 · Why the web-service instinct breaks

Start with the shape Kubernetes already understands. A stateless web pod is cheap to start (milliseconds), holds no important state in memory, serves a request in one round trip, and is bottlenecked on CPU — so an HPA that watches CPU and adds replicas is exactly the right control loop. Every one of those four properties is false for an LLM server.

A GenAI service carries a model artifact: the trained weights, plus a tokenizer, config, and license. For a large model this artifact is tens to hundreds of gigabytes, and it must be fully present in accelerator memory before the server can answer a single request. So "ready" is no longer "the process bound to the port"; it is "the weights finished loading," which can take minutes. And the request itself is not homogeneous. It arrives as a prompt, the server first reads the whole prompt, then emits output one token at a time, streaming each token as it is produced. That is two phases with completely different resource profiles, glued to a long-lived stream.

→artifact readiness → weights load from registry/storage into GPU memory (seconds to minutes), once per pod start

→prefill → read the entire prompt in one compute-heavy pass, build the KV cache

→decode → emit output tokens one at a time, each step re-reading the KV cache (memory-bandwidth-heavy)

→stream + policy → tokens flow back to the user as they are produced, under safety/guardrail checks

The mental model to carry through the whole track: a GenAI platform is a token factory. Kubernetes schedules containers, but the real product is bounded latency per generated token under scarce GPU, memory, storage, and policy constraints. The container is just the crate the factory ships in. Everything difficult — cold starts measured in minutes, autoscaling on signals CPU cannot see, packing concurrent requests into a fixed pool of accelerator memory — comes from that factory, not from the crate.

2 · The LLM fundamentals you actually need

The book's LLM fundamentals are operational, not academic. You need exactly enough to reason about the platform; here is that floor, each term defined at first use.

Tokenization

Text is split into tokens (sub-word units; very roughly ~0.75 words each in English). Tokens are the work units: cost and latency scale in tokens, not characters. A "context window" is a token budget.

Embeddings + transformer layers

Each token becomes a vector (an embedding); the model is a stack of transformer layers that attend over all prior tokens. This is why work grows with sequence length — every token attends to every earlier one.

Prefill (compute-heavy)

Processing the input prompt in a single forward pass over all prompt tokens at once. It is dominated by matrix-multiply throughput — it saturates the GPU's compute units. Long prompts make prefill expensive.

Decode (bandwidth-heavy)

Generating output one token at a time. Each step does little compute but must re-read the growing KV cache from memory, so it is bound by memory bandwidth, not compute. This is why one slow stream can't be sped up by "more cores."

The object that ties prefill and decode together is the KV cache (key-value cache): during prefill the model computes, for every token and every layer, a key and value vector used by attention, and caches them so that each decode step does not recompute the whole history. The KV cache lives in HBM (High-Bandwidth Memory — the fast on-package memory of a GPU, e.g. 80 GB on an A100/H100-class accelerator), right alongside the weights. It grows with every token generated and with every concurrent request. That single fact — weights and KV cache share the same scarce 80 GB — is the source of most GPU-memory pressure in the rest of this track. (The serving runtime's job is largely to pack that cache efficiently; the vLLM KV cache lesson goes deep on the mechanism, and ML Systems Design serving covers the broader serving picture.)

Finally, two latency numbers that matter more than "request latency," because they are what a streaming user actually feels:

→TTFT (time-to-first-token) → how long from request arrival until the first output token appears. Dominated by queueing + prefill. This is the "did it freeze?" feeling.

→TPOT (time-per-output-token) → the steady-state gap between successive tokens during decode. This is the "reading speed" feeling; its inverse is tokens/sec per stream.

A 4-second total response can feel instant (200 ms TTFT, then a smooth stream) or broken (3.8 s of nothing, then a dump). Average request latency hides that difference entirely. If you remember one thing: a request is not one homogeneous HTTP transaction — it is artifact-readiness + prefill + many decode steps + streaming + policy.

3 · Worked example: a 70B model on an 80 GB GPU

Make it concrete. Take a 70-billion-parameter model — a common open-weight size. In fp16 (16-bit floats, 2 bytes per parameter) the weights alone are 70 × 10⁹ × 2 bytes ≈ 140 GB. That does not fit on a single 80 GB GPU at all, so even before serving you face a platform decision: shard across GPUs (covered later in the track) or quantize (store weights at lower precision). At 4-bit (0.5 byte/param) the same model is 70 × 0.5 ≈ 35 GB — now it fits on one 80 GB accelerator, leaving roughly 80 − 35 = 45 GB of HBM for everything else, chiefly the KV cache. (Quantization is not free — it can cost some output quality — but it is the cheapest way to make a big model fit one card.)

That leftover HBM is your concurrency budget. Suppose the KV cache costs, for this model, on the order of ~0.15 MB per token (an illustrative figure; it scales with layers × KV heads × head dimension × 2, and is larger for models without grouped-query attention). Then a single 8,000-token context occupies 8,000 × 0.15 MB = 1,200 MB ≈ 1.2 GB, and your ~45 GB of free HBM holds roughly 45 / 1.2 ≈ 38 such requests at once — your true max concurrency, set by memory, not CPU. Halve the context to 4,000 tokens (≈0.6 GB each) and you fit 45 / 0.6 ≈ 75. This is the curve the widget below lets you drag.

HBM occupancy — weights + KV cache vs. concurrency

The bar is one 80 GB GPU's HBM. The dark block is the model weights (fixed once you pick a precision); the green block is the KV cache, which grows with concurrent requests × context length. Drag the knobs: when green + dark exceeds the bar, you are out of memory — the platform must reject requests, evict cache, or add a GPU. Notice that CPU never appears here. That is the point.

precision: concurrent requests: 20 context tokens: 8000

Weights

—

KV cache

—

Free / over

—

Max concurrency at this context

—

Now watch the web-service instincts fail, one by one, against this single GPU:

A CPU HPA cannot see the bottleneck. During decode the GPU is busy but the host CPU is near-idle, and the real limit — free HBM for KV cache — is invisible to CPU metrics. Scaling on CPU either never fires (so you silently saturate HBM and queue or drop requests) or fires on the wrong signal. You need GPU/serving metrics (TTFT, queue depth, KV-cache utilization) instead.
A normal readiness probe lies. A default probe checks that the process answers on its port — which happens in milliseconds, long before the ~40 GB of weights have finished loading into HBM (which takes from tens of seconds to minutes). Kubernetes marks the pod Ready, sends it traffic, and every request fails or hangs until load completes. Startup needs a real model-loaded signal and generous startup probes.
TTFT is invisible to request-count autoscaling. You can be well under your nominal request limit yet have TTFT climbing because prefill is queueing behind long prompts. The number users feel (first token) is not the number CPU or RPS expose.

4 · The first trade-off, and the four numbers

The first decision is not "which Kubernetes object should I use?" It is "why am I running this model myself?" Running an LLM in-cluster means owning GPUs, drivers, model registries, scaling policy, safety reviews, and incident response — real, expensive operational surface. So the answer had better be a reason that buys product leverage:

Privacy / compliance

Data cannot leave your boundary (PHI, regulated records). The model must run where the data already lives.

Data gravity / latency

The model sits next to large in-cluster data or low-latency services; a round-trip to an external API would dominate.

Control

Custom routing, fine-tunes/LoRA adapters, specific model versions, deterministic behavior — things a hosted API will not give you.

Cost at steady scale

At high, predictable volume, owning GPUs can beat per-token API pricing — but only once utilization is high. "Because we can" is not on this list.

If your reason is one of these, Kubernetes is an attractive substrate: it gives a common control plane for artifacts, accelerators, routing, autoscaling, observability, safety, and cost. But the control plane only works when the workload exposes the right signals — which is the whole job of the rest of this track. The payoff for accepting the burden is control.

Once you have decided to self-host, there is one design habit to install before drawing any architecture. Write down four numbers:

The four numbers	What it pins down	Routes you to a…
Expected prompt tokens	Prefill cost and KV-cache size per request; artifact/context-length needs	storage / GPU-memory problem
Expected output tokens	Decode duration and therefore how long a stream holds HBM	GPU-memory / throughput problem
Target TTFT	How much queueing + prefill you can tolerate before the first token	routing / scheduling problem
Target concurrent requests	How many KV caches must coexist in HBM at once	GPU-count / cost problem

Those four numbers, run through the arithmetic in §3, tell you which scarce resource binds first — and that is the only thing that should drive your initial architecture. Below is the same pressure-to-signal mapping the rest of the track keeps returning to.

Pressure	What it changes	Operational signal to watch
Model size	Startup time, storage layout, rollout speed, node-local cache strategy	model-load seconds, image/weights cache hit rate
KV cache	Achievable concurrency and HBM pressure (the §3 budget)	active tokens in cache, KV-cache utilization %
Streaming tokens	User-perceived latency — what actually feels fast or broken	TTFT and TPOT (p50/p99), not just request latency
Safety policy	Prompt, tool, and output controls layered onto the stream	blocked actions, eval regressions, audit trail completeness

5 · Failure modes and the platform's job

Failure modes

LLM-server-as-CPU-web-pod. Scaling only on CPU or request count. During decode the GPU is the bottleneck and CPU is idle, so the autoscaler either never fires while HBM saturates, or fires on a signal unrelated to the real limit. Signal: flat CPU while TTFT and queue depth climb.
Ignoring artifact startup. Treating a 10 GB–800 GB weights load as instant. A default readiness probe passes in milliseconds and routes traffic to a pod whose weights are still loading. Signal: a burst of errors/timeouts immediately after every scale-up or rollout.
Optimizing the wrong latency. Driving down average request latency while TTFT or p99 inter-token latency — the numbers users actually feel on a stream — silently regress. Signal: good mean latency dashboards, but users report "it freezes" or "it stutters."
Forgetting the shared-HBM constraint. Sizing GPUs for the weights only, then being surprised when concurrency collapses because the KV cache had nowhere to live. Signal: requests rejected or evicted under load despite "the model fits."

Implementation checklist

Define serving SLOs first: TTFT, TPOT, throughput (tokens/s), error rate, and quality guardrails — because these, not CPU/RPS, are what you will scale and alert on.
Name the scarce resource before designing: HBM, GPU count, interconnect bandwidth, storage bandwidth, or external-data access. The §3 arithmetic tells you which binds first.
Decide what must stay in-cluster for privacy, compliance, or latency — that is the actual justification for owning GPUs at all.
Keep LLM fundamentals short and operational: tokenization, prefill, decode, KV cache, model-artifact size — only as deep as the platform decision needs, no more.
Write the four numbers (prompt tokens, output tokens, target TTFT, target concurrency) on every new workload before any YAML exists.

Checkpoint exercise

Try it

Take a workload you might self-host (say, an internal coding assistant). Write the four numbers — expected prompt tokens, expected output tokens, target TTFT, target concurrency. Pick a model size and precision, and using the §3 arithmetic estimate (a) whether the weights fit one 80 GB GPU, and (b) how many concurrent requests the leftover HBM holds at your context length. Then state, in one sentence, which scarce resource binds first — and which web-service instinct (CPU HPA, default readiness probe, or average-latency dashboards) would have hidden that bottleneck from you.

Where this points next

You now have the frame: a GenAI service is a token factory whose binding constraint is artifact readiness and shared HBM, not CPU, and whose true SLOs are TTFT and TPOT, not request latency. The natural next question is "so how do I actually get one running?" Lesson 2, First Manual LLM Deployment, takes the concepts here and makes them physical: it stands up a real LLM server on Kubernetes by hand — pulling weights, picking a serving runtime, wiring a probe that actually waits for the model to load, and requesting a GPU — so you feel each of this lesson's failure modes before any automation hides them. Everything after that (autoscaling, routing, multi-tenancy) is machinery layered on top of that first hand-built pod.

Takeaway

A GenAI platform is a token factory: Kubernetes schedules the container, but the product is bounded latency per generated token under scarce GPU, HBM, storage, and policy constraints. An LLM request is not one HTTP transaction — it is artifact-readiness + prefill (compute-heavy) + many decode steps (bandwidth-heavy, re-reading the KV cache) + streaming + policy — so the web-service instincts break: a CPU HPA cannot see the HBM/TTFT bottleneck, a default readiness probe routes traffic to a pod whose weights have not loaded, and average latency hides the TTFT/TPOT a streaming user actually feels. The worked example pins it down: a 70B model is ~140 GB in fp16 (needs sharding or 4-bit quantization to ~35 GB to fit an 80 GB GPU), and the leftover ~45 GB of HBM — shared with the weights — is your real concurrency budget. The first trade-off is not "which Kubernetes object?" but "why am I running this myself?" (privacy, data gravity, control, cost at scale), and the four numbers — prompt tokens, output tokens, target TTFT, target concurrency — route you to a storage, GPU-memory, routing, or cost problem.

Interview prompts

Why can't you treat an LLM server like a stateless web pod? (§1 — it carries a heavy artifact that must load before it can answer, its request is a two-phase stream (prefill + many decode steps) not one transaction, and its bottleneck is HBM/bandwidth, not CPU — so CPU-based autoscaling and default readiness probes both misfire.)
Define prefill vs. decode and why their resource profiles differ. (§2 — prefill processes the whole prompt in one pass and is compute-heavy; decode emits one token at a time, each step re-reading the growing KV cache, so it is memory-bandwidth-heavy. More compute cores won't speed a single decode stream.)
What is the KV cache and why does it constrain concurrency? (§2, §3 — cached per-token key/value vectors in HBM that let decode skip recomputing history; it grows with tokens × concurrent requests and shares the same 80 GB HBM as the weights, so leftover HBM after weights sets max concurrency.)
Why are TTFT and TPOT better SLOs than average request latency? (§2 — they are what a streaming user feels (time to first token, then reading speed); the same total latency can feel instant or frozen, and the mean hides which.)
Walk the 70B-on-80GB arithmetic. (§3 — 70B × 2 bytes ≈ 140 GB fp16 (won't fit one GPU → shard or quantize); 4-bit ≈ 35 GB fits, leaving ~45 GB HBM for KV cache; at ~0.15 MB/token an 8k context is ~1.2 GB, so ~38 concurrent requests — memory, not CPU, sets concurrency.)
What is the real first trade-off in a GenAI platform? (§4 — not "which Kubernetes object?" but "why am I running this model myself?" Justified only by privacy/compliance, data gravity/latency, control, or cost at steady scale; the payoff for the operational burden is control.)
What are the "four numbers" and what do they decide? (§4 — expected prompt tokens, expected output tokens, target TTFT, target concurrent requests; run through the §3 arithmetic they tell you whether you face a storage, GPU-memory, routing, or cost problem — i.e. which scarce resource binds first.)