Part I - Inference foundations

First Manual LLM Deployment

Lesson 01 argued that GenAI workloads break the assumptions Kubernetes was tuned for: the pods are huge, they pin scarce accelerators, and they take minutes to become useful. That was the why. This lesson is the how, stripped of every convenience: you deploy one LLM server by hand — a Deployment, a Service, a GPU request, a model mount, probes — and send it a first request. The point is not that you will run production this way; you will not. The point is that KServe, Ray Serve, Helm charts, and platform APIs are all automation wrapped around exactly these objects, and you cannot debug a wrapper whose primitives you have never assembled yourself.

1 · The contract a serving pod must satisfy

Before any framework, an inference service is a pod with five obligations. It must run a runtime image — the server binary plus the CUDA stack (the NVIDIA libraries and kernels the GPU code links against). It must obtain the model weights from somewhere. It must be scheduled onto a node that actually has a GPU, which means requesting one explicitly. It must expose a port for inference requests and, in practice, a second one for metrics. And it must tell Kubernetes when it is ready to receive traffic. Higher-level tools hide each of these, but none of them removes the obligation — they just decide it for you.

The single idea this whole lesson turns on is the last one, and it is where engineers coming from ordinary web services get burned. A normal web server is ready the moment its process binds a port — the handler is in memory, there is nothing to warm. An LLM server is different: it binds its port in a second or two, then spends minutes doing real work before it can answer anything — reading tens of gigabytes of weights off disk, copying them into GPU memory (HBM, the high-bandwidth memory soldered onto the accelerator, distinct from the host's system RAM), compiling or autotuning CUDA kernels (CUDA is NVIDIA's GPU programming stack; the server JIT-compiles or selects the optimal kernel variants for the specific GPU model on the node), building the KV-cache allocator (the KV cache is the buffer that stores the attention keys and values for tokens already processed, so the model does not recompute them on every decode step; the allocator carves up leftover HBM into pages for it), and loading any LoRA adapters (LoRA — Low-Rank Adaptation — produces small fine-tuned weight deltas that are merged onto or served alongside the base model). "Process up" and "model ready" are separated by a multi-minute gap, and confusing the two is the defining failure of manual LLM deployment.

It is worth making "model ready" physically concrete, because it is not one event but a sequence, and each stage can be the thing that hangs. A useful mental model is a four-stage warm-up, each with its own dominant cost:

Only after stage 4 can the server honestly answer a request inside its normal latency. Stages 1–4 are exactly the work that a TCP-port check is blind to.

2 · Making readiness honest with probes

Kubernetes gives you three probes, and an LLM pod needs all three doing distinct jobs. A probe is a periodic health check Kubernetes runs against the container — here, an HTTP GET against a path, where any 2xx/3xx status counts as a pass. The three differ only in what a pass or fail does:

The mistake is using a liveness-style probe (or a short readiness probe) to cover the cold start: the model has not finished loading inside the deadline, Kubernetes concludes the container is dead, kills it, and the replica enters a restart loop that never completes a load. The startupProbe exists precisely to break that loop — it is the only probe whose job is to say "still loading, leave it alone." Because it suppresses the liveness probe, you are then free to make the liveness deadline tight (catch a truly hung server fast) without it ever firing during the legitimate multi-minute warm-up.

readinessProbe:
  httpGet:
    path: /v1/models        # endpoint returns 200 only once the model is loaded
    port: 8000
  periodSeconds: 10
  failureThreshold: 3       # 3 consecutive fails -> pulled from Service endpoints

startupProbe:
  httpGet:
    path: /health           # cheap "process is alive and loading" check
    port: 8000
  periodSeconds: 10
  failureThreshold: 60      # tolerate up to 60 x 10s = 600s of model loading

The arithmetic is the point. A 600-second startup budget (60 × 10s) is not arbitrary padding — it must exceed your worst realistic load time, which §4 lets you estimate. Set failureThreshold too low and a slow node or a cold volume kills the pod mid-load; set it absurdly high and a genuinely hung load wastes a GPU for an hour before anyone notices. The /v1/models path in the readiness probe matters too: a server like vLLM only returns the model in that list after weights are resident in HBM, so checking it — rather than a generic /health — is what makes readiness mean "can answer."

3 · A minimal, production-minded manifest

Here is the whole serving path as plain objects. It is deliberately not abstracted: model path declared explicitly, GPU requested explicitly, runtime version pinned, the inference/metrics port exposed, probes matched to load time. (vLLM, like most OpenAI-compatible servers, serves both the /v1/* inference API and the Prometheus /metrics scrape endpoint on the same port, so one containerPort covers both — a separate metrics port is only needed for servers that split them.) Read it as the assembly diagram every later controller hides.

apiVersion: apps/v1
kind: Deployment
metadata: { name: llm-server }
spec:
  replicas: 1
  selector: { matchLabels: { app: llm-server } }
  template:
    metadata:
      labels: { app: llm-server }
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.11.0         # PIN an exact runtime + CUDA stack; never :latest
        args: ["--model", "/models/llama-3-8b"]  # serve from the mounted path, not a hub id
        ports:
        - { name: http, containerPort: 8000 }    # vLLM serves /v1/* AND /metrics on this one port
        resources:
          limits:
            nvidia.com/gpu: 1                     # GPUs go under limits; K8s mirrors it into requests
        volumeMounts:
        - { name: model-store, mountPath: /models, readOnly: true }
        readinessProbe:
          httpGet: { path: /v1/models, port: 8000 }
          periodSeconds: 10
          failureThreshold: 3
        startupProbe:
          httpGet: { path: /health, port: 8000 }
          periodSeconds: 10
          failureThreshold: 60                    # ~600s load budget (see §4)
        livenessProbe:
          httpGet: { path: /health, port: 8000 }  # suppressed until startupProbe passes
          periodSeconds: 10
          failureThreshold: 3                      # tight: restart a TRULY wedged process fast
      volumes:
      - name: model-store
        persistentVolumeClaim: { claimName: llm-weights }  # PVC holds the weights
---
apiVersion: v1
kind: Service
metadata: { name: llm-server }
spec:
  selector: { app: llm-server }
  ports:
  - { name: http, port: 80, targetPort: 8000 }    # narrow contract: one inference port

And the first request, once the pod is Ready — this is the moment that proves the whole path:

kubectl port-forward svc/llm-server 8080:80
curl localhost:8080/v1/completions \
  -d '{"model":"/models/llama-3-8b","prompt":"Hello","max_tokens":16}'

Notice what the manifest does not do: no autoscaling, no canary, no traffic splitting, no model versioning. Those are precisely the jobs the controllers in later lessons add. By building the path without them, you learn that a controller's failure is always a failure to perform one of these plain responsibilities — it scheduled a pod that never got a GPU, or it flipped readiness before the model loaded. The manifest is your debugging baseline.

4 · How the weights reach the pod — with the number that decides it

The most consequential design choice is also the least visible: where the model bytes come from and when. There are four paths, and they trade deployable size, promotion speed, and — above all — cold-start time, the seconds between a new replica being scheduled and it being able to answer. Cold start is the GenAI tax: every scale-up event pays it, so it directly bounds how fast you can absorb a traffic spike.

This is also why §3 puts weights on a PVC and pins the runtime image separately: you can patch a CVE in the server without re-shipping 30 GB of weights, and you can roll a new model version without rebuilding the server. Coupling them — baking weights into the runtime image — is the convenient demo choice that quietly makes every promotion a multi-gigabyte event.

5 · A failure taxonomy — classify before you edit

When a manual deploy fails, the operational habit that pays off for the entire rest of the track is to classify the failure before changing any YAML. A broken serving pod falls into exactly one of six buckets, and the bucket tells you which signal to read:

Six labels, one decision tree, and three commands resolve it in order — never guess, read the signal:

Internalize this and the rest of the track — controllers, operators, autoscalers — becomes "which of these six did the automation get wrong?" A KServe InferenceService stuck not-ready is the same readiness-vs-network question, just one layer up; an autoscaler that adds replicas which never serve is the cold-start arithmetic of §4 colliding with a too-short probe budget.

Checkpoint exercise

Where this points next

You now have one working serving path and a vocabulary for why it breaks. But doing this by hand for every model, version, and rollout does not scale — and the readiness, GPU-request, and weight-delivery logic you just wrote longhand is exactly what a controller should own. Lesson 03, Model Servers and Controllers, introduces the model-server runtimes (vLLM and friends) and the Kubernetes controllers that wrap them, so that "deploy a model" becomes a single declarative spec instead of the manifest above. Every field that controller exposes — and every default it picks — maps back to a line you just wrote here, which is the whole reason we built it by hand first.

Interview prompts

• Why is an LLM server's readiness fundamentally different from a web server's? (§1 — a web server is ready when it binds a port; an LLM server binds in seconds but then spends minutes loading weights into HBM, compiling kernels, and warming caches, so readiness must mean "model loaded and can answer," not "socket open.")
• What three probes does an LLM pod need and what does each do? (§2 — startupProbe tolerates the long load and suppresses the others; readinessProbe gates Service traffic on a model-loaded check like /v1/models; livenessProbe restarts a wedged process — and must not be set shorter than the load time or it kills slow loads.)
• Walk through every obligation a serving pod has, and where each lives in the manifest. (§1, §3 — pinned runtime image, model-weight source, explicit GPU request, inference + metrics ports, and an honest readiness signal; the manifest maps one field to each.)
• Compute the cold start for a 30 GB model three ways and pick one. (§4 — ~240 s init-download over 1 Gbit/s (30×8/1), ~30 s off a 1 GB/s PVC read, near-instant from a warm baked 35 GB image; choose by which bottleneck — egress, registry pull, or storage read — you can afford under your worst scale-up burst.)
• A pod is Ready but curl times out — how do you classify it? (§5 — network-exposure vs. readiness: check whether the Service has endpoints and whether readiness truly means model-loaded; the six-bucket taxonomy plus kubectl get pods/endpoints and logs localize it before any YAML edit.)
• Why package the runtime image separately from the weights? (§3, §4 — so a server CVE patch does not re-ship 30 GB and a model bump does not rebuild the server; coupling them makes every promotion a multi-gigabyte rebuild-and-repush.)
• What is the single most common cause of "traffic errors right after a scale-up"? (§2, §5 — readiness flipping green before weights finish loading, so the Service routes to a cold runtime; fix by probing a model-loaded endpoint and budgeting the startupProbe above the computed cold-start time.)