System ML — Parallel Training & Inference, From First Principles

A linearized tour of how a 70B-parameter model is trained on thousands of GPUs and served to millions of users. Each lesson isolates one mechanism — derived from a bandwidth-vs-compute argument, not memorized from a slide deck.

The system you're learning

Every distributed deep-learning system reduces to one tension: HBM, compute, and interconnect are three orthogonal resources, and the way you partition the work decides which one binds first. The seven parallelism strategies below are seven different bets about which axis is cheap and which is scarce on your hardware. The art is composing them so all three are saturated at once.

Each strategy lives at a different point on the triangle. Pure DDP sits squarely in the centre (it spends HBM and interconnect, not much else). FSDP slides toward the interconnect corner — trading communication for memory. TP and EP push almost everything onto the interconnect. The lessons walk this triangle.

The four questions this series answers

1. Why can't I just buy a bigger GPU? Because the model, the gradients, the optimizer state, and the activations grow at different rates than HBM does. Memory is the first wall (lesson 01).
2. Why is AllReduce a constant — not a function of N? Ring topology. The bandwidth-optimal collective derivation in lesson 02 is the single most useful piece of math in this series.
3. Why does TP "want" to stay on one node? The bandwidth pyramid: 10 TB/s registers, 3 TB/s HBM, 0.9 TB/s NVLink, 0.05 TB/s IB per NIC. Two orders of magnitude per layer (lesson 03).
4. What changes between training and inference? Training is dominated by gradient sync; inference is dominated by per-step memory bandwidth. The same hardware, used differently. Lessons 14–15.

Part I · Foundations (lessons 01–03 · the language and the constraints)

Part II · Training parallelism (lessons 04–09 · six strategies, six places to shard)

Each strategy shards something different: the data, the optimizer state, the weights themselves, the layers, the sequence, or the experts. Read in order and you can derive any modern training stack (Megatron-LM, DeepSpeed, NeMo, FSDP) as a composition of these six.

                replicate                           shard
                weights                              optimizer
        ┌── DDP ───────────────────── FSDP / ZeRO ──┐ memory
        │  (lesson 04)                  (lesson 05) │
        │                                            │
        │ shard within                shard across   │
        │   layers                       layers      │
        ├── TP ──────────────────────── PP ──────────┤ compute
        │  (lesson 06)                  (lesson 07)  │
        │                                            │
        │ shard the                   shard the      │
        │ sequence                    experts        │
        └── SP / CP ────────────────── EP ───────────┘ specialty
           (lesson 08)                 (lesson 09)

Part III · Training memory & batch (lessons 10–11 · the other two HBM consumers, and the batch math)

Parts I–II shard the parameters, gradients, and optimizer state across ranks. But two more quantities grow with a real run — the activations and the effective batch — and neither is fixed by sharding weights. These two lessons close lesson 01's memory accounting and set up the batch math before we compose everything in Part IV — which is exactly why they sit here, ahead of the kernel stack: lessons 01, 05, and 12 already lean on activation memory.

Part IV · Composition, overlap & inference (lessons 12–15 · putting it together, hiding the comms, and serve-time)

Part V · The single-GPU stack (lessons 16–22 · framework, kernels, compilers)

Parts I–IV sit at the cluster level: ranks, collectives, fabric. Below all that is one GPU running one forward pass — and the per-GPU throughput is set by a stack of layers most of which you never see. The Python you write becomes an autograd graph, which becomes a dispatcher trace, which becomes a stream of kernel launches, which become matmuls and elementwise ops, which read and write HBM. Each layer is a place where performance leaks or is reclaimed.

  user code  ────▶  PyTorch dispatcher  ────▶  autograd graph
  (lesson 16)         (lesson 16)                (lesson 16)
                                                       │
                              ┌────────────────────────┘
                              ▼
                       autocast / precision  ─────▶  caching allocator
                          (lesson 17)                  (lesson 18)
                              │                            │
                              ▼                            │
                       kernel launch  ◀──────  CUDA stream │
                          (lesson 19)                      │
                              │                            │
                              ▼                            │
                  ┌── handwritten CUDA / cuBLAS / cuDNN ───┤
                  ├── Triton  (lesson 20)                  │
                  └── Inductor codegen  (lesson 21)        │
                                                           │
                            CUDA Graphs / TensorRT  (lesson 22)
                                       │
                                       ▼
                                   HBM

Part VI · Running the job (lessons 23–26 · measure it, keep it stable, feed it, survive it)

Parts I–V give you every mechanism and how to make one GPU fast. But a real training run is a three-week, multi-thousand-GPU job: you have to see the bottleneck you were taught to predict, keep the loss from diverging, stay fed with data, and survive the hardware dying under it. These four lessons are the cross-cutting concerns every large run actually spends its engineering on.

Beyond this track · CUDA kernels & serving engines

The CUDA-primitives track that once lived here — GPU execution model, memory hierarchy, vector add, coalesced access, tiled matmul, warps & divergence, reductions, occupancy, tensor cores — is now the foundations half of the GPU Kernels for ML Engineers track, where it sits directly under the serving lessons that depend on it (FlashAttention, paged KV, prefix reuse, scheduling, framework).

How to read this

1. Linearly. Lesson n assumes the math of n−1. The collective derivation in 02 is reused in 04, 05, 06, and 09; the bandwidth pyramid from 03 explains every "why intra-node?" sentence in Part II.
2. Touch every widget. Each one has a regime where the strategy fails. Find it. The failure is the lesson — "what breaks DDP" tells you why FSDP exists; "what blows up TP" tells you why TP stays on one node.
3. Do the back-of-envelope. The numbers in this series are not decorative. Memorize four: NVLink ~900 GB/s peak (≈150 GB/s effective on a flat ring), IB ~50 GB/s/NIC, HBM ~3.35 TB/s, H100 peak ~1 PFLOP bf16. Most distributed-training arguments are settled by comparing two of these.