Part I - Inference foundations

Model Servers and Controllers

In lesson 02 you stood up a single LLM by hand: a Deployment that ran a serving process, a Service in front of it, and a readiness probe so traffic only arrived once the model had loaded. It worked, but every decision was implicit — what runs the tokens, what owns the rollout, what knows the model is healthy. As soon as you have two models, two teams, and a canary to roll out, those decisions stop being implicit and start being architecture. This lesson names the three layers that were tangled together in that one manual deployment — the model server, the inference controller, and the programmable serving framework — and shows that they are not competitors on a shortlist but distinct kinds of ownership. Confusing them is the most common way a serving platform ends up both rigid and impossible to debug.

1 · Three layers, three kinds of ownership

Every LLM endpoint you will ever run answers HTTP and returns tokens. That shared surface is exactly why people flatten these tools into one comparison — "should we use vLLM or KServe or Ray Serve?" — as if picking one. That question is malformed. They sit at different layers and a serious platform often runs all three at once: vLLM inside a pod, a KServe controller managing that pod, and Ray Serve composing a multi-step app that calls it. The useful question is never "which one" but "which kind of ownership is the hard part of my problem."

An analogy that keeps the layers apart: the model server is the engine — it converts fuel into motion and nothing else, but it must do so efficiently. The controller is the fleet manager — it does not turn a single crankshaft; it decides how many vehicles run, swaps them in and out safely, and reports which are healthy. The framework is the dispatcher writing custom routes in code — it composes several engines and stops into one journey. An engine cannot manage a fleet; a fleet manager cannot improve combustion. When a platform asks one layer to do another's job, that is where it breaks.

2 · The server layer: what makes a model server, not a Flask app

In lesson 02 the Deployment ran "a serving process." That process was a model server, and it is doing far more than loading weights and calling model.generate() in a request handler. The reason a purpose-built server (vLLM, Hugging Face's TGI, or SGLang) exists at all is that naive serving wastes most of an expensive GPU. The runtime mechanisms below are the difference, defined at first use:

vLLM is the cleanest mental anchor because it makes these concerns explicit — PagedAttention for KV-cache memory, continuous batching for scheduling — and exposes an OpenAI-compatible HTTP endpoint, so clients written against the OpenAI API work unchanged. TGI occupies the same model-server category with its own operational trade-offs and production hardening. SGLang pushes the unit of work toward programs with structured outputs and aggressive prefix reuse. The cross-track lesson SGLang vs vLLM compares those two in depth.

The point of the worked number is not the exact figures — they shift with model, hardware, and traffic shape — but the location of the lever. Continuous batching, KV cache management, and quantization are server concerns. No amount of controller or framework sophistication recovers a 16× gap you left on the table inside the runtime. That is the first half of avoiding the most common failure mode: reaching for a controller to fix a problem that lives one layer down.

3 · The controller layer: one InferenceService instead of the lesson-02 stack

A model server is a process. Running one of them by hand was lesson 02. The trouble starts at scale: every team that wants an endpoint now repeats your Deployment, your Service, your probe, your autoscaler, your rollout dance — and each does it slightly differently, so the platform team cannot reason about any of them uniformly. An inference controller solves this the Kubernetes way: it adds a custom resource (a CRD) that describes the intent — "serve this model with this runtime" — and a control loop that reconciles the lower-level objects to match. KServe is the standard example.

Concretely, here is the manual stack from lesson 02 — a Deployment running vLLM, a Service, and a readiness probe — written out the long way:

# lesson 02, by hand: you own every object and every field
apiVersion: apps/v1
kind: Deployment
metadata: { name: llama-13b }
spec:
  replicas: 2
  template:
    spec:
      containers:
        - name: vllm
          image: vllm/vllm-openai:latest
          args: ["--model", "meta-llama/Llama-13b", "--port", "8000"]
          resources: { limits: { nvidia.com/gpu: 1 } }
          readinessProbe: { httpGet: { path: /health, port: 8000 } }
---
apiVersion: v1
kind: Service
metadata: { name: llama-13b }
spec: { selector: { app: llama-13b }, ports: [{ port: 80, targetPort: 8000 }] }
# ...plus an HPA, an Ingress, and a hand-rolled rollout when the model changes.

KServe collapses all of that into one declaration. You state the model and the runtime; the controller creates and owns the Deployment, the Service, the routing, the revisions, the autoscaling, and the status — and gives the platform team one uniform object to govern across every team:

# with a controller: you state intent, KServe reconciles the rest
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata: { name: llama-13b }
spec:
  predictor:
    model:
      modelFormat: { name: vllm }      # which runtime owns token execution
      storageUri: "s3://models/llama-13b"
      resources: { limits: { nvidia.com/gpu: 1 } }
    minReplicas: 2                       # autoscaling, revisions, rollout,
    maxReplicas: 8                       # networking, and status are now the
                                         # controller's job, not yours.

The lesson 02 objects did not disappear — KServe still produces a Deployment, a Service, and probes underneath. They became an implementation detail the controller owns. That is the whole value: a new revision rolls out by changing one field; the platform sets policy once (resource quotas, allowed runtimes, autoscaling defaults) and every team inherits it; and the controller's status reports readiness uniformly. KServe is Kubernetes-native in feel — you think in declarative resources, the platform reconciles them — which fits exactly when you want a standardized, platform-owned serving API and users mainly supply model and runtime configuration. (Autoscaling and request-aware routing for these objects is its own subject, covered later in this track.)

4 · The framework layer: composition in Python, and the KServe vs Ray Serve line

A controller is excellent when the unit of work is "an endpoint serving a model." But some applications are not one model behind one endpoint — they are a graph: preprocess, embed, retrieve, call a small model, then a big one, post-process, fan results back together. Expressing that as a pile of separate InferenceService objects wired by network calls is awkward, because the composition itself is the application logic, and it wants to live in code. A programmable serving framework owns that. Ray Serve (running on a Ray cluster, which on Kubernetes is provisioned by the KubeRay operator) lets you declare deployments as Python objects and compose them, while Ray manages the distributed workers underneath:

import ray
from ray import serve

@serve.deployment(num_replicas=2, ray_actor_options={"num_gpus": 1})
class Generator:
    def __init__(self):
        self.llm = load_vllm("meta-llama/Llama-13b")   # the server still owns tokens
    async def __call__(self, prompt: str) -> str:
        return await self.llm.generate(prompt)

@serve.deployment
class Pipeline:
    def __init__(self, gen):
        self.gen = gen                                  # composition lives in code
    async def __call__(self, req):
        ctx = await retrieve(req.query)                 # a RAG step, in Python
        return await self.gen.remote(build_prompt(ctx, req.query))

app = Pipeline.bind(Generator.bind())                   # wire the graph, then `serve.run`

Notice the layers are still distinct: inside Generator a real model server (vLLM here) still owns token execution. Ray Serve is not competing with vLLM — it is composing it. What Ray adds is Python-native composition plus distributed execution: the framework decides which worker runs which deployment, scales replicas, and handles the calls between them. That fits when application logic, custom multi-step composition, or data-science ownership of the serving code matters more than a narrow, declarative inference CRD.

The sharp contrast — and the question interviewers actually ask — is KServe versus Ray Serve:

Treating these as interchangeable because both expose HTTP is the third failure mode. They differ in who owns the serving logic: a declarative platform contract versus a Python program. The bridge between a raw GPU runtime and these higher-level frameworks is explored further in the GPU serving framework bridge.

5 · The decision rule: pick by the hard part, keep boundaries explicit

Because all three expose HTTP, the temptation is to pick by familiarity. Pick by where the difficulty lives instead:

And whichever you adopt, keep the boundaries explicit. The controller must not obscure runtime metrics — you should still be able to read vLLM's own throughput and KV-cache occupancy without parsing reconcile logs. The runtime must not become the organization's policy system — rollout and quota are the controller's job, not a server flag. The framework must not silently re-implement lifecycle the platform already standardized. Layers that bleed into each other produce a platform that is simultaneously rigid (you cannot change one layer without the other) and hard to debug (no layer owns a clean answer).

6 · Failure modes and a checklist

Checkpoint exercise

Where this points next

This lesson assumed the model weights were simply there — the KServe spec waved at storageUri: "s3://models/llama-13b" and the Ray deployment called load_vllm(...) as if loading a 26 GB artifact were free. It is not. Where the weights live, how they are versioned, how a pod pulls tens of gigabytes onto a GPU node before it can become ready, and how a controller decides which artifact a revision points at are their own discipline. Lesson 04, Model Artifacts and Registries, adds that machinery: model storage, registries, versioning, and the pull path that feeds the server layer you just learned to own.

Interview prompts

• Why is "vLLM vs KServe vs Ray Serve" a malformed question? (§1 — they sit at different layers — token execution, lifecycle, composition — and a real platform runs all three together; the right question is which kind of ownership is the hard part.)
• What does a model server own that a Flask wrapper around generate() does not? (§2 — prefill/decode scheduling, KV-cache management, continuous batching, prefix caching, quantized KV, speculative decoding, and LoRA hot-swapping; these are where GPU efficiency lives and yield order-of-magnitude throughput gains.)
• What exactly does a KServe InferenceService replace from a hand-built deployment? (§3 — the Deployment, Service, readiness probe, autoscaler, routing, and rollout become implementation details the controller reconciles from one declared intent, governed uniformly by the platform team.)
• Contrast KServe and Ray Serve. (§4 — KServe is Kubernetes-native: declare an InferenceService, the platform owns the API and reconciles lifecycle; Ray Serve is Python-native: write and compose deployments in code, the application team owns composition and Ray manages distributed workers. A server still owns tokens inside either.)
• How do you decide which layer to reach for? (§5 — per-token efficiency → fix the runtime; many teams safely deploying endpoints → put a controller first; composing a multi-step Python app across workers → evaluate a framework.)
• A team adopts KServe to cut latency and it does not help. What went wrong? (§2, §6 — the bottleneck was a runtime concern (no continuous batching, undersized KV cache); a controller owns lifecycle, not token execution, and cannot recover throughput lost in the server.)
• Why keep an escape hatch to a raw Deployment even after adopting a controller? (§6 — to reproduce and isolate runtime bugs without the full controller path, and to keep server observability and tuning accessible when the abstraction misbehaves.)