Part II - GPU platform
Multi-GPU Inference and Topology
Lesson 06 turned bare GPU machines into a schedulable platform layer: nodes labeled by accelerator type, the device plugin advertising nvidia.com/gpu, taints and node pools that keep the right pods on the right silicon. That all assumed the unit of serving is one GPU. But the moment a model's weights do not fit in one GPU's memory — or one GPU cannot serve enough concurrent users — you have to spread the work across several. This lesson is about the three fundamentally different ways to use more than one GPU, why they are not interchangeable, and why the physical wiring between those GPUs decides which one is even allowed.
New capability: Decide when to replicate a model, shard it across GPUs on one node, or distribute it across nodes — and predict how each choice moves the bottleneck onto memory, compute, or the interconnect.
1 · One GPU works — now what?
By lesson 06 you can put a model server in a pod, ask for one GPU, and have Kubernetes land it on the right node. The next question is forced on you by one of two limits. Either the model is too big — its weights exceed a single GPU's memory — or one replica is too slow — a single GPU cannot keep up with the request volume at your latency target. These two limits have completely different answers, and conflating them is the most common multi-GPU mistake.
Two terms used throughout, defined now. HBM (high-bandwidth memory) is the fast on-package memory soldered to a GPU — an NVIDIA A100 has 80GB, an H100 80GB, an H200 141GB. It is where weights, the KV cache, and activations all have to live; when people say "fits on the GPU" they mean "fits in HBM." The KV cache is the per-request store of attention keys and values for every token already generated; it grows with sequence length and with the number of concurrent requests, and it competes with the weights for the same HBM. Keep both in mind — they reappear in the capacity math.
The mental model for the whole lesson:
The decision rule that falls out of this: replicate until you cannot, then shard inside the fastest interconnect domain, and cross a node boundary only when forced. The rest of the lesson is making each clause precise.
2 · The three parallelism strategies
"Use more GPUs" is not one technique. The book lists three, and they differ in what they split and therefore what they cost.
The distinction that matters most: replication never communicates during inference; tensor and pipeline parallel do. A replica is an island. A tensor-parallel group is a single organism whose neurons are wired together — and the wire's speed becomes part of every token's latency. That is why the same GPUs can serve beautifully as four replicas and terribly as one TP-4 group spanning a slow link: the strategy, not the hardware, decides whether the interconnect is on the critical path.
3 · Does it fit? The capacity math
Whether you can replicate is a memory question, and you answer it with arithmetic, not intuition. The dominant term is the weights. A model in fp16 (2 bytes per parameter) needs roughly 2 × params bytes:
But "the weights fit" is the wrong stopping point, because weights are not the only thing in HBM. You must budget four pools separately:
| Memory pool | What it is | Scales with |
|---|---|---|
| Weights | The model parameters, loaded once and shared by all requests | Model size and precision (fixed) |
| KV cache | Per-request attention keys/values for generated tokens | Concurrency × sequence length (grows) |
| Activation scratch | Transient buffers for the current forward pass | Batch size, hidden dim |
| Communication buffers | Staging for all-reduce / pipeline transfers (TP/PP only) | Shard count, message size |
4 · Topology — the wire decides what is allowed
Kubernetes decides which node a pod lands on. It does not, by default, know or care about the physical graph of links between GPUs — and that graph is what determines whether tensor parallel is fast or fatal. (NVLink is NVIDIA's direct GPU-to-GPU link; NVSwitch is the on-board crossbar that wires every GPU in a node to every other at full NVLink speed, so an 8-GPU node behaves as one all-to-all fabric rather than a chain.) The interconnect hierarchy, fastest to slowest:
| Link | Where | Rough bandwidth | Good for |
|---|---|---|---|
| NVLink / NVSwitch | GPUs inside one node | ~900 GB/s (NVLink 4) | Tensor parallel — the all-reduce is cheap enough to do every layer |
| InfiniBand | GPUs across nodes (HPC fabric) | ~25-50 GB/s | Pipeline parallel across nodes; barely-tolerable TP if you must |
| Standard Ethernet | Generic cloud networking | ~1-12 GB/s | Replication only — never put TP collectives over this |
Why this matters so much: tensor parallel makes every token a communication event. Each layer ends in an all-reduce across the TP group, and a model has dozens of layers, and you generate hundreds of tokens per response. If that all-reduce travels over NVLink at ~900 GB/s it is invisible; if it travels over Ethernet at ~10 GB/s it dominates, and your per-token latency — TPOT, time per output token — collapses by an order of magnitude. A TP group that spans a slow link does not "run slower" — it stops being viable.
So the placement rule is concrete: keep a tensor-parallel group entirely within one NVLink domain (one node, where the GPUs are NVSwitch-connected). On Kubernetes you enforce this with the same tools from lesson 06, now aimed at locality rather than just type:
# Pin a TP-8 server to a single 8-GPU node so all-reduce stays on NVLink.
# Request all 8 GPUs in one pod -> the scheduler must place it on one node.
resources:
limits:
nvidia.com/gpu: 8 # whole node; TP group cannot be split across hosts
# And steer to nodes that actually have NVSwitch:
nodeSelector:
gpu.topology/nvlink-domain: "full" # a label your node setup applies
For multinode (pipeline parallel across hosts, or TP-then-replicate layouts), you additionally want the cooperating pods placed close in the network — same rack, same InfiniBand partition. Pod affinity with a topology key keeps cooperating pods together; the gang-scheduling concern (all-or-nothing placement for a distributed group) is what the next lessons build toward. The point for now: topology-aware placement is a deliberate act. Left alone, the scheduler will happily pack two unrelated GPU pods onto the host you wanted whole, breaking the NVLink locality your TP group assumed.
5 · Choosing — the trade table
Putting it together, here is the decision as a table you can run top-to-bottom:
| Strategy | Buys | Costs | When |
|---|---|---|---|
| Replication | Throughput, fault isolation, simplicity (no collectives) | Duplicates weights and warm KV caches across copies | Model fits one GPU — the default; scale out, not up |
| Tensor parallel | Model fit when weights exceed one GPU's HBM | All-reduce every layer, every token; needs NVLink | Weights > one GPU but ≤ one node's combined HBM |
| Pipeline parallel | Fit for models too big for even one node's TP group | Pipeline bubbles; latency and utilization hit | Model exceeds a single node; combine with TP within stages |
| Multinode | Capacity beyond a single host | Slower collectives over the network; larger failure surface | Forced by size; keep TP inside nodes, PP/replication across |
6 · Where placement goes wrong
Failure modes
- TP across a slow link. A tensor-parallel group is placed across nodes (or onto GPUs joined only by PCIe/Ethernet), so every layer's all-reduce crosses a slow boundary and every token pays. The signal: TPOT is 5-20× worse than a single-node TP run on the same model; network counters spike in lockstep with generation.
- The scheduler packs your "whole" node. You assumed eight free GPUs on one host for a TP-8 group, but the platform placed an unrelated GPU pod there first, breaking NVLink locality or leaving the group split. The signal: intermittent latency depending on which node won the placement race; the fix is requesting the whole node or using affinity/anti-affinity.
- KV cache ignored in capacity math. You sized GPUs so the weights just fit, declared victory, and shipped. Under real concurrency the KV cache exhausts HBM, requests queue or OOM, and compute sits idle. The signal: out-of-memory errors or a hard concurrency ceiling well below what the compute could handle.
- Sharding a model that fits. A 13B model put behind TP-2 "to use the GPUs," adding all-reduce latency to every token for zero fit benefit when two independent replicas would have served more, faster. The signal: lower throughput and higher latency than the same GPUs running as replicas.
Implementation checklist
- Did you compute weights = bytes/param × params and check it against one GPU's HBM before reaching for TP? Replicate if it fits.
- Is every tensor-parallel group confined to a single NVLink/NVSwitch domain (one node), never split across a slower link?
- Are you using node selectors, whole-node GPU requests, pod affinity, or topology labels to keep cooperating shards physically close?
- Did you budget the four memory pools separately — weights, KV cache, activation scratch, communication buffers — and size KV cache against target concurrency?
- For multinode, is the slow boundary on the least chatty axis (replication or pipeline stages), with the chatty TP collectives kept inside nodes?
Checkpoint exercise
Where this points next
You can now reason your way to a layout — replicate the 13B, TP-2 the 70B inside a node, keep the collectives on NVLink. But a layout chosen by arithmetic is a hypothesis, not a result. The next lesson, 08 - Benchmarking and Runtime Tuning, gives you the machinery to test it: how to measure TTFT (time to first token), TPOT, and throughput under realistic load, how to find the batch size and concurrency where latency knees upward, and how to tell whether the bottleneck your placement created is memory, compute, or — as this lesson warned — the interconnect. Placement decides what is possible; benchmarking tells you what actually happened.
Interview prompts
- What are the three ways to use more than one GPU, and what does each buy? (§2 — replication: throughput + fault isolation, no communication; tensor parallel: model fit, all-reduce per layer/token; pipeline parallel: fit for very large models, at the cost of bubbles.)
- A 70B model is fp16. Does it fit one 80GB GPU, and what do you do? (§3 — 70B × 2 bytes ≈ 140GB > 80GB, so no; you need TP ≥ 2. TP-2 puts ~70GB weights per GPU, leaving ~10GB for KV cache.)
- A 13B model fits one GPU — should you ever shard it across two? (§2, §3, §6 — no; sharding a model that fits only adds all-reduce latency to every token for zero fit benefit. Replicate it; two replicas serve more, faster.)
- Why is tensor parallel so sensitive to the interconnect? (§4 — TP all-reduces at the boundary of every layer for every token, so on NVLink it is invisible but on Ethernet it dominates; a TP group spanning a slow link stops being viable, not just slower.)
- You sized GPUs so the weights fit. Why might concurrency still be tiny? (§3 — the KV cache shares the same HBM and grows with concurrency × sequence length; once the leftover after weights is exhausted, requests queue or OOM regardless of idle compute.)
- How do you keep a tensor-parallel group from being split across a slow link on Kubernetes? (§4 — confine it to one NVLink domain: request the whole node's GPUs in one pod, use node selectors/topology labels for NVSwitch nodes, and affinity to co-locate cooperating pods.)
- For a model that spans multiple nodes, which communication should cross the node boundary? (§4, §5, §6 — the least chatty axis: replication or pipeline stages over InfiniBand/Ethernet, while the per-token TP collectives stay inside nodes on NVLink.)