Part VI - Governance and capstone

Capstone: Production GenAI Platform

Lesson 17 was the last new mechanism: agents that loop, call tools, and hold state on the cluster. You now have every part — artifacts and registries, weight delivery, the GPU platform, multi-GPU topology, runtime tuning, autoscaling and routing, disaggregated serving, observability, guardrails, customization, training jobs, batch scheduling, RAG, and agents. What you do not yet have is the wiring that lets all of it change safely while serving live traffic. This capstone assembles the seventeen lessons into one production blueprint and gives that blueprint its missing organ: a control loop that proposes a change, proves it is safe, promotes it, watches it, and reverses it when the watch goes red.

1 · A platform is a control loop, not a deployment

The instinct after sixteen lessons of mechanism is to think the job is "run vLLM in a pod and point a Service at it." That is a deployment, and a deployment is a frozen snapshot. A platform is the opposite of frozen: it is a system whose entire purpose is to change — new model versions ship weekly, prompts get edited, adapters get swapped, traffic shifts — while never dropping below its quality, latency, and cost guarantees. The only structure that makes constant change safe is a feedback loop, the same shape as a thermostat or a PID controller: propose a change → measure its effect → keep it or reverse it → feed what you learned back into the rules.

Three terms define the loop. TTFT (time to first token) is the latency from request to the first output token — it is dominated by the prefill phase, where the model reads the prompt and fills its KV cache. TPOT (time per output token) is the steady-state inter-token latency during decode, when the model emits one token at a time. $/1M tokens is the unit cost of serving — section 4 derives it. These three plus error rate are the SLOs the "watch" stage guards; if a release moves any of them the wrong way, the loop reverses it automatically.

The mental model: routing (lesson 09) chose which one branch handles a request; this loop chooses which version of the whole stack is live, and it does so continuously rather than once. The deployment is just the loop's current set-point — the temperature the thermostat is currently holding. Push the set-point (propose a change), and the loop's job is to reach it without ever letting the room get uncomfortable.

Each stage maps to a piece of machinery you already built, which is why this is a capstone and not a new topic. The propose stage is a declarative diff (a registry digest, a routing weight, an adapter) reconciled by GitOps — section 3. Evaluate reuses the offline eval harness from guardrails (lesson 12) and the benchmark contracts from tuning (lesson 08). Promote is the weighted router from autoscaling/routing (lesson 09). Watch is the metric/trace pipeline from observability (lesson 11). Reverse is a routing-weight flip — cheap precisely because the incumbent never stopped running. And feed back is the discipline that turns each incident into a permanent eval case or policy rule, so the loop's rules tighten over time instead of decaying. A loop missing any one stage is not a slower loop — it is an open loop, which is to say a bet.

2 · The full platform map

Every lesson in this track is a box in one diagram. The hot path (left to right) is what a user's request actually traverses; the surrounding planes — registry, observability, policy, cost, and the eval-feedback edge — are the control loop wrapped around it.

                         ┌──────────── control plane ────────────┐
   git (source of truth)  │  registry/signing(04)  policy/RBAC(12,§5) │
        │ GitOps reconcile │  observability(11)     FinOps/quota(§4)   │
        v                  └───────────────┬───────────────────────┘
  ┌─────────────┐    ┌──────────────┐    ┌──┴────────────┐    ┌────────┐
  │ model        │    │ GPU runtime   │    │ router/gateway │    │ users  │
  │ artifact(04) │ -> │ vLLM on GPUs  │ -> │ autoscale +    │ -> │        │
  │ weights(05)  │    │ topology(07)  │    │ canary(09)     │    │        │
  │              │    │ tuning(08)    │    │ guardrails(12) │    │        │
  └─────────────┘    │ disagg(10)    │    └───────────────┘    └────┬───┘
   adapters(13)  ->   │ + RAG(16)     │      ^                        │
   training jobs(14)  │ + agents(17)  │      │ release gates          │ feedback
   batch/fairness(15) └──────────────┘      │ <- incidents <----------┘
                                            (cost/security/eval failures)

Read the map as flows, not just boxes. A model enters as a signed artifact (lesson 04), its weights are delivered into pods (lesson 05), it runs on a GPU node (lesson 06) across some tensor-parallel or pipeline topology (lesson 07) tuned for throughput (lesson 08), possibly disaggregated into separate prefill and decode pools (lesson 10), and adapted with LoRA adapters or fine-tuning jobs (lessons 13–14) run under batch/fairness scheduling (lesson 15). Tokens flow through the router and gateway, autoscaled and canaried (lesson 09), filtered by guardrails (lesson 12). Identity rides every hop as RBAC and tenant context. State lives in the KV cache, RAG index (lesson 16), and agent memory (lesson 17). Evidence — traces, metrics, eval scores — pours into observability (lesson 11). And money is metered as GPU-hours and dollars per million tokens (section 4). The control loop is the act of changing any box and letting the surrounding planes tell you whether the change was good.

3 · One release, end to end

Concrete is the only way this lands. Walk a single release of a chat model from "a new checkpoint exists" to "100% of traffic is on it" — or back to where it started. GitOps is the substrate: the desired state of the platform (model references, routing weights, replica counts, policies) lives declaratively in a git repository, and a reconciler (Argo CD / Flux) continuously drives the cluster to match what git says. You do not kubectl apply to production; you open a pull request, and merging it is the release. This matters for the loop: git is the audit log of every set-point change, rollback is a git revert, and "what is deployed" is answerable by reading a file instead of interrogating the cluster.

The change that starts the release is a one-line diff to the routing config — a canary (a small slice of real traffic sent to the new version, named after the caged bird miners used as an early warning):

# platform/routing.yaml — the desired state, in git
model: chat-70b
candidate:
  ref: sha256:c3d4...            # new build, pinned by digest, not :latest
  weight: 5                       # canary: 5% of traffic
incumbent:
  ref: sha256:a1b2...            # still running, takes the other 95%
  weight: 95

The canary's rollback criteria are defined before rollout and encoded as alerts, e.g.:

# rollback if ANY of these fire on the canary during the bake window
canary_rollback_if:
  - p95_ttft_ms        > 1.15 * baseline_p95_ttft_ms   # >15% TTFT regression
  - p95_tpot_ms        > 1.15 * baseline_p95_tpot_ms
  - error_rate         > 0.5%                          # absolute ceiling
  - quality_score      < baseline_quality - 0.02       # eval regression
  - cost_per_1m_tokens > 1.20 * baseline_cost          # FinOps SLO (§4)

4 · FinOps — cost is an SLO, with a real number

FinOps is making GPU spend visible and controllable in the same terms as latency and quality. The instinct is to treat a cost spike as a finance problem discovered at month-end. On a GPU platform that is far too late: idle GPUs and runaway token volume burn money by the hour. So cost becomes a platform SLO with its own alert (it was in the rollback list above), and the unit you reason in is dollars per 1M tokens.

The controls that defend this SLO are the same ones from autoscaling and scheduling, re-read as cost levers:

Track these per workload: $/GPU-hour, $/1M input tokens and $/1M output tokens (output is the expensive half — it is the decode phase), idle-GPU cost, cold-start waste, KV-cache/prefix-cache hit savings, and the metric that ties cost to value: cost per successful task. A cheap model that fails half its tasks is not cheap.

Attribute every one of those numbers to a tenant. Once each team's traffic carries an identity (the same identity RBAC uses in section 5), GPU-seconds and tokens can be metered per team and turned into chargeback (the team's budget is actually billed) or showback (they see the cost without being billed). This closes the FinOps loop: a team that owns its $/1M-token line will tune its own prompts and pick a smaller model when it can, in a way no central mandate achieves. The per-tenant quota from lesson 15 is then both a fairness control and a spend cap — the same mechanism reread through the cost lens.

5 · Security across the full path

The most common security mistake on a GenAI platform is securing the application — the pod, the Service, the API — while leaving the things that make it a model platform unguarded. The threat surface is the whole path: model artifacts, registries, prompts, retrieved documents (lesson 16), tool calls (lesson 17), adapters (lesson 13), training data (lesson 14), logs, and human-approval flows. Each is a place a secret can leak, a poisoned input can enter, or an unsafe action can fire.

Four of those terms earn a definition, because they are where GenAI platforms differ from ordinary web services. Pod Security Admission (PSA) is Kubernetes' built-in admission controller that enforces the Pod Security Standards: label a namespace restricted and the API server rejects any pod that runs as root, mounts the host filesystem, or asks for privileged capabilities — exactly the escape hatches a compromised model server would reach for. Artifact signing (Sigstore/cosign) attaches a cryptographic signature to a model or image; an admission policy then refuses to run anything whose signature does not verify, so an unsigned or tampered model never reaches a GPU. An SBOM (software bill of materials) is a machine-readable inventory of every dependency baked into an artifact — the manifest you grep when a CVE drops, instead of guessing which of 400 images is affected. Tenant isolation is the guarantee that team A's prompts, retrieved documents (lesson 16), KV cache, and logs can never be seen by team B — enforced by namespaces, per-tenant ServiceAccounts and NetworkPolicies, and per-tenant RAG indices, not by hoping the application filters correctly.

The platform principle that ties them together: make the secure path the easy path. If shipping a signed, SBOM-attested model through the GitOps pipeline is less work than side-loading an unsigned one, engineers do the safe thing by default — the policy gate is the path of least resistance, not a hurdle. Security that relies on people choosing the harder route fails the first busy week. The full upstream reference is the Kubernetes security checklist.

6 · Incident drills — closing the loop

The loop is only real if the "feed back" edge works: every incident must become a permanent test or policy so the same failure cannot ship twice. Map each incident class to its fast control and its durable follow-up artifact.

Run these as drills, not only as real incidents: deliberately deploy a slightly-worse model and confirm the eval gate catches it; inject latency and confirm the canary rolls back inside the bake window. A rollback path you have never exercised is a rollback path that does not work.

Checkpoint exercise

Where this points next

This is the last lesson — there is no next mechanism to add, because you now have all of them and the loop that wires them together. The hand-off is to practice: take a real workload and walk it around the loop until the muscle is reflexive. Two directions help. Re-read any component lesson — 08 tuning, 09 autoscaling/routing, 12 guardrails — now seeing each as a knob inside the release loop rather than a standalone topic, and return to the track index to see the whole arc at once. And for the layer above this one — how the platform serves a product, not just a model — the ML Systems Design capstone picks up where this leaves off. Then go browse the rest of all_lessons: the platform you built here is the substrate every other track's system runs on.

Interview prompts

• Why is a production GenAI platform a control loop rather than a deployment? (§1 — a deployment is a frozen snapshot, but a platform's purpose is constant change; only a feedback loop — propose, evaluate, promote, watch, reverse, feed back — makes that change safe while holding SLOs.)
• Walk a single model release end to end. (§3 — PR bumps the registry digest → offline eval gate → shadow traffic → 5% canary watching TTFT/TPOT/error/quality/cost → ramp 5/25/50/100% or instantly roll back by shifting the routing weight to the still-running incumbent.)
• What makes rollback fast, and why is canary-with-rollback better than a hard cutover? (§3 — the incumbent keeps serving 95% during the canary, so rollback is just a routing-weight change in seconds; the escape hatch is already open, vs a cutover where the old version must be redeployed.)
• Derive $/1M tokens and name the biggest lever to lower it. (§4 — GPU node price ÷ token throughput; 8×H100 at $20/hr ÷ 4.32M tok/hr ≈ $4.63/1M. The lever is throughput at fixed cost — better batching/tuning (lesson 08) and avoiding idle GPUs, since utilization scales cost inversely.)
• Why treat cost as an SLO instead of a finance report? (§4 — GPUs burn money by the hour; a month-end review catches runaway tokens or idle nodes weeks late. Wiring a $/1M-token threshold into the canary's rollback criteria catches cost regressions at deploy time, defended by quota/admission/load-shedding.)
• What does "secure the full path" mean beyond securing the pod? (§5 — the threat surface includes model artifacts, registries, prompts, retrieved docs, tool calls, adapters, and training data; controls layer RBAC, default-deny NetworkPolicy, restricted Pod Security, external secrets, signing/SBOMs/license checks, PII redaction, and tenant isolation — and the secure path must be the easy path.)
• How does an incident close the loop? (§6 — each incident class maps to a fast control (canary rollback, eval halt, quota/shed, policy block) and a durable follow-up (new benchmark, eval case, capacity model, guardrail test); drill the rollback so it actually works when needed.)