Part II - GPU platform

GPU Nodes as a Platform Layer

Lesson 05 got the model weights into the pod — the multi-gigabyte tensors are now sitting on a volume the container can read. But weights are inert without a chip to run them on, and a freshly provisioned GPU node does not announce itself to Kubernetes the way a CPU does. Plain Kubernetes counts CPU and memory natively; it has no built-in idea of what a GPU is, how many a node has, or whether one is already in use. This lesson is about the platform plumbing that turns a bare GPU box into a node the scheduler can target — and the sharing modes that decide whether one physical card serves one tenant or seven. Get this layer wrong and your pod lands on a GPU node, starts, and then fails the moment it asks for CUDA.

1 · The GPU is a stack, not a count

When you write resources: { limits: { cpu: "2" } }, Kubernetes already understands CPU intimately — the kubelet measures it, cgroups enforce it, the scheduler bin-packs it. A GPU gets none of this for free. The kernel does not know how to talk to the card without a vendor driver; the container runtime does not know how to expose the device files; and the Kubernetes scheduler has no field called "GPU" at all. Every one of those gaps is filled by a separate piece of software, and if any layer is missing the symptom is the same maddening one: the pod schedules fine and then cannot find a GPU.

So the mental model to carry through this lesson is: a GPU is a stack. Driver, runtime, device plugin (or DRA driver), node labels, topology, isolation mode, telemetry, and scheduling policy — each is a distinct layer, and "the GPU works" means all of them are present and agree. The book's Chapter 3 walks this stack precisely because most "my GPU pod doesn't work" incidents are platform incidents, not model incidents. The fix lives below the model server, not in it.

2 · The platform stack, layer by layer

Here is the full pipeline a request for a GPU traverses, from silicon to a process that can call CUDA. Each stage is a real component you install or rely on; the names matter because each is a separate failure point.

Two of these names deserve a sharper definition at first use. Node Feature Discovery (NFD) is a Kubernetes add-on that detects hardware and system features on each node and exposes them as node labels; GPU Feature Discovery (GFD) is the GPU-specific companion that adds labels like nvidia.com/gpu.product=H100, nvidia.com/gpu.memory=81920, and nvidia.com/mig.strategy. A device plugin is the kubelet extension mechanism (a gRPC service) by which a vendor advertises non-CPU/memory resources; NVIDIA's device plugin is what makes nvidia.com/gpu a thing the scheduler can count and allocate.

Installing and version-matching all six layers by hand on every node is exactly the "snowflake node" trap. The NVIDIA GPU Operator exists to collapse it: it is a Kubernetes Operator that installs and lifecycle-manages the driver, container toolkit, device plugin, NFD/GFD, and DCGM telemetry (move 5) from a single cluster-level configuration. The platform lesson is that this stack is code — a declarative policy you version and review — not a checklist someone runs once over SSH and forgets.

3 · The resource model — and what a single number hides

With the device plugin in place, requesting a GPU looks deceptively like requesting CPU. Here is the canonical whole-GPU pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: llm-server
spec:
  containers:
  - name: vllm
    image: vllm/vllm-openai:latest
    resources:
      limits:
        nvidia.com/gpu: 1        # whole GPU; this pod owns the device

Note that GPUs are requested only as a limit, and the count is an integer — you cannot ask for "0.5 GPU" through the basic device plugin. The pod that gets nvidia.com/gpu: 1 owns the entire physical device with full memory and compute isolation. Simple, but the scheduler made its decision knowing only the integer 1. It did not know whether that GPU has 24 GB or 80 GB of HBM, nor whether the node's two GPUs are NVLink-connected. A 70B-parameter model in 16-bit needs roughly 70 × 2 = 140 GB of weights plus KV cache — it will not fit on one 80 GB card, but nvidia.com/gpu: 1 happily schedules it anyway, and the pod OOMs at load time. The number was the wrong unit (move 1) made operational.

Hardware partitioning gives the scheduler richer units. With MIG (Multi-Instance GPU) — NVIDIA's feature that splits one physical GPU into hardware-isolated slices — a single A100 can be carved into up to seven instances, each with its own dedicated memory and compute. Those slices are advertised as named resources, so a pod requests a profile:

    resources:
      limits:
        nvidia.com/mig-1g.10gb: 1   # one MIG slice: ~1/7 compute, 10 GB HBM

Now the resource name itself carries memory and compute size (1g.10gb = one compute slice, 10 GB), so the scheduler is no longer flying blind on capacity. That is the difference: nvidia.com/gpu: 1 says "a GPU, somewhere"; nvidia.com/mig-1g.10gb: 1 says "10 GB of isolated GPU." Topology — which GPUs are NVLink-adjacent — is still not in either request, which is the whole subject of lesson 07.

4 · Sharing modes — utilization against isolation

One H100 can cost on the order of $2–$4 / GPU-hour on a cloud, so leaving it 15% utilized by a small model is burning money. Sharing modes exist to raise utilization — but every form of sharing trades away some isolation, and isolation is the assumption your tenants are quietly relying on. The four modes, defined at first use:

The sharpest distinction to internalize: MIG isolates, time slicing does not. MIG draws hardware fences — separate memory controllers, separate compute partitions — so a misbehaving tenant in one slice physically cannot touch another's memory or steal its cycles. Time slicing just rotates the GPU's attention between processes that all see the same memory; it improves average utilization but provides zero protection, so a single pod that allocates too much HBM can crash every other pod on the device. Use time slicing for cooperative, low-stakes, bursty workloads; never use it where one tenant's bug must not become another tenant's outage.

const HBM = 80, PER = 16, SLICE = 20, MIG_MAX = 4;
// time slicing: one shared pool, OOM if demand exceeds capacity
const tsMem = N * PER;
const tsOOM = tsMem > HBM;                 // no isolation -> overcommit kills pods
// MIG: hard slices, scheduler refuses beyond capacity
const migPlaced = Math.min(N, MIG_MAX);    // each tenant fenced into ~20 GB
// MIG tenants are always isolated; time-slice tenants never are.

DRA deserves an honest maturity note. DRA (Dynamic Resource Allocation) generalizes the rigid "count of named resources" model into ResourceClaims: a pod describes the device it wants (a GPU with at least 40 GB, or a specific sharing arrangement) and a DRA driver matches and allocates it, much like a PersistentVolumeClaim does for storage. It is the strategic direction for accelerators on Kubernetes, and the core API has graduated into stable Kubernetes releases (the v1.34–v1.36 line moved DRA forward substantially). But "the API is stable" is not "every vendor driver and every advanced sharing feature is production-ready in your cluster." In 2026 the practical readiness questions are still vendor driver support, scheduler integration, telemetry, and whether your on-call engineers know how to debug a ResourceClaim that fails to allocate. Adopt it deliberately; do not overstate it.

5 · Where this breaks and what to install

Checkpoint exercise

Where this points next

This lesson made a single GPU schedulable and decided how many tenants share it. But the resource model still hides the thing that dominates large-model performance: how multiple GPUs are wired together. A 70B model that does not fit on one card must be split across several, and whether those cards share NVLink or only PCIe changes throughput by an order of magnitude — yet nvidia.com/gpu: 2 says nothing about it. Lesson 07, Multi-GPU Inference and Topology, adds the machinery the scheduler needs to place co-dependent GPUs correctly: tensor parallelism across a node, topology-aware scheduling, and gang scheduling so all the ranks of one model land together or not at all.

Interview prompts

• Why doesn't Kubernetes understand a GPU the way it understands a CPU? (§1, §2 — CPU/memory are native and cgroup-enforced; a GPU needs a vendor driver, container toolkit, discovery, and a device plugin/DRA driver to become a schedulable resource, so any missing layer means the pod schedules but cannot see the device.)
• Walk the stack from hardware to a process calling CUDA. (§2 — hardware → driver+runtime/container toolkit → discovery (NFD/GFD labels type/count/driver/MIG/topology) → device plugin or DRA driver → scheduler resource model → pod device visibility.)
• What does nvidia.com/gpu: 1 hide, and why does it matter? (§1, §3 — it loses HBM size, GPU generation, isolation mode, and topology; a 140 GB model schedules onto an 80 GB card and OOMs, or tensor-parallel ranks land on non-NVLinked GPUs.)
• Contrast MIG and time slicing precisely. (§4 — MIG hardware-partitions a GPU into isolated slices with dedicated memory/compute (real isolation); time slicing software-interleaves processes over one shared memory pool (no isolation), so a tenant can OOM or add jitter to neighbors.)
• What is DRA and how mature is it in 2026? (§4 — Dynamic Resource Allocation: claim-based ResourceClaims describing the device needed, allocated by a DRA driver; the core API is stable in recent Kubernetes but vendor-driver and feature readiness is still uneven — beta/maturing, don't overstate.)
• What does the NVIDIA GPU Operator do, and why call its config "platform code"? (§2 — it installs and lifecycle-manages driver, toolkit, device plugin, NFD/GFD, and DCGM from one declarative cluster policy, replacing hand-built snowflake nodes and preventing driver/runtime version skew.)
• A GPU pod is Running but the model server can't find CUDA. How do you debug? (§5 — bottom-up: node sees device → driver matches runtime → discovery labeled it → plugin/DRA advertised it → scheduler placed it on a compatible node → container sees the device file; debug the model server last.)