Communication–computation overlap

Every collective in this series is dead time — unless it hides behind compute. This is the one principle behind DDP bucketing, FSDP prefetch, async-TP, and zero-bubble pipelines. We've used it piecemeal; here we name it, derive it, and find its limit.

The one equation

A distributed training step does two kinds of work. It runs compute kernels (matmuls, attention, the optimizer), and it runs communication (NCCL collectives moving gradients, parameters, or activations between GPUs). The naive assumption is that these are serial — you compute, then you communicate, then you compute again — so wall-clock is their sum:

But compute and communication use different hardware paths. The SMs (streaming multiprocessors) run matmuls; the copy engines and the NVLink/IB fabric move bytes. There is no fundamental reason a GPU can't grind a matmul on its SMs while NCCL pushes a previous layer's gradient out over NVLink. When they run at the same time, the communication is hidden — it costs zero wall-clock. One caveat the "different paths" framing hides: NCCL collective kernels also occupy SMs to drive the transfer, so even off the critical path, overlap contends with compute for SM resources — which is why achieved overlap never quite reaches the theoretical maximum. The real equation is:

where T_overlap is how much of the communication actually ran concurrently with compute. The quantity we care about — the thing that shows up as a stall on your profiler — is the leftover:

This is exposed communication: the part of a collective that no compute was available to hide behind, so it sits on the wall-clock as pure overhead. Two limiting cases make the stakes concrete:

• Zero overlap (T_overlap = 0): we're back to the naive sum. Every byte of communication is exposed. This is what a careless implementation gets.
• Perfect overlap (T_overlap = min(T_compute, T_comm)): the two streams run fully in parallel and the step takes as long as the slower of the two: T_step = max(T_compute, T_comm). If you're compute-bound (T_comm ≤ T_compute), communication is completely free.

The entire engineering game of distributed training is driving T_exposed toward zero. Recall lesson 04's number: a 7B model's gradient AllReduce moves ~28 GB per rank on the wire (≈2× the 14 GB bf16 gradient buffer, for a ring) and takes ~190 ms on a 150 GB/s flat ring, against a ~100 ms backward. Serial, that's a step time of ~290 ms — the AllReduce nearly triples it. Hidden, the step is max(100, 190) = 190 ms; and if you can shrink the comm under 100 ms (hierarchical AllReduce, lesson 02), it vanishes entirely. Same bytes on the wire. The difference is whether they overlap.

The mechanism: CUDA streams

Overlap is not automatic. It is a property of how kernels are scheduled on the GPU, and the unit of scheduling is the CUDA stream. A stream is an ordered queue of work for the GPU: kernels enqueued on the same stream run one after another, in order. Kernels on different streams have no ordering constraint between them — the GPU's scheduler is free to run them concurrently if it has spare resources.

In PyTorch, your forward and backward kernels land on the default (current) stream. NCCL collectives, by default, run on their own separate communication stream created by the process group. So the two are already on different streams — the hardware can overlap them. Whether it does depends entirely on the dependency structure:

1. Input-ready. A collective on the comm stream can only start once its input tensor has been produced by the compute stream. NCCL inserts a cross-stream wait (a CUDA event) so the AllReduce of layer k's gradient does not begin until that gradient kernel has finished. You cannot communicate a value before it exists.
2. Consumer-later. For the overlap to be useful, the result of the collective must be consumed later than the compute it overlaps with. If the very next kernel on the compute stream needs the collective's output, the compute stream blocks on it and there is nothing to hide behind.

That second condition is the whole story. The hardware bandwidth is rarely the limiter on whether overlap happens — the dependency graph is. If a collective sits between two compute steps that both depend on it, no amount of fast NVLink will hide it; it is on the critical path by construction. If the collective's output is needed only much later, even a slow collective hides perfectly. Restructuring computation to create that slack is the art.

compute stream:  [grad L=k] [grad L=k-1] [grad L=k-2] ...
                     |  event signals "grad k ready"
                     v
comm stream:         [AllReduce grad k ............]   <- runs concurrently
                                                       with grad k-1, k-2 compute

The mechanics of which stream a kernel lands on, how events synchronize them, and how the caching allocator keeps per-stream memory from colliding are a lesson of their own — see lesson 18 (caching allocator & streams). For now the model is enough: compute on one stream, comm on another, overlap is gated by the dependency between them.

Case study — DDP: bucketed AllReduce (recall lesson 04)

DDP is the cleanest place overlap pays off, because backward hands you a natural source of slack. Backward propagates from the loss toward the input, so gradients materialize in reverse layer order: layer L's gradient first, then L-1, …, then layer 0. The moment a layer's gradient exists, its AllReduce can fire on the comm stream while the compute stream keeps grinding earlier layers' gradients. Each AllReduce hides behind all the backward compute that comes after it.

Two details matter. First, you don't fire one AllReduce per layer — NCCL has a per-launch latency floor (tens of microseconds), and on a 100-layer model that's milliseconds of pure latency tax before any bytes flow. So DDP buckets: consecutive layers' gradients are packed into a flat buffer, and the bucket is AllReduced as one message when it fills. This is exactly lesson 02's α/β cost model in action:

The bucket size is a genuine tradeoff with a U-shaped cost:

• Too small → many buckets, each paying the α launch latency; you live in the latency-bound regime where ring AllReduce is inefficient. Total comm time balloons.
• Too big → a bucket only fires after enough layers have filled it, so the first AllReduce starts late in backward and there's less compute left to hide it behind. In the limit of one giant bucket, you AllReduce only after all of backward — zero overlap, back to the naive sum.

PyTorch's default is bucket_cap_mb = 25 MB, a reasonable middle for typical layer sizes and NVLink/IB bandwidths. Widget 2 below makes the U-shape visible.

Second — and this is the part people miss — there is an irreducible exposed tail. The bucket containing the first layers' gradients (layers near the input) is the last to fill, because those gradients are produced last in backward. It fires when backward is already done. There is no compute left after it. So that final bucket's AllReduce is fully exposed:

You can shrink it (smaller final bucket → less to expose) but you cannot eliminate it: backward must finish before the last gradient exists, and that gradient still has to be reduced. This tail is precisely what keeps DDP scaling efficiency below 100% — 95% is excellent, and the missing 5% is mostly this tail plus the launch tax.

Case study — FSDP: prefetched AllGather (recall lesson 05)

FSDP (ZeRO-3) shards every layer's parameters across ranks, so before you can compute layer k in the forward pass you must AllGather its full parameter shard back together. Naively that's a serial stall in front of every layer: gather params, compute, discard, gather next. The collective sits directly in front of the compute that needs it — the worst possible dependency.

The fix is prefetch, and it's the mirror image of DDP. While the compute stream is busy on layer k, the comm stream runs the AllGather for layer k+1 ahead of time. By the time compute finishes layer k, layer k+1's parameters are already assembled and waiting. The gather hides behind the previous layer's matmul. Backward does the same in reverse, and adds a ReduceScatter of gradients (the FSDP analogue of DDP's AllReduce) that overlaps the same way.

Prefetching costs memory: to gather layer k+1 while computing layer k, you must hold both layers' full (unsharded) parameters at once. Prefetch two layers ahead and you hold three. PyTorch's limit_all_gathers caps the number of in-flight AllGathers — it is the dial trading peak memory against overlap depth. Set it too low and the gather can't get far enough ahead to hide; too high and you OOM. This is the same memory-vs-overlap tension as DDP's bucket size, wearing a different hat.

Case study — TP: the AllReduce you cannot hide (recall lesson 06)

Tensor parallelism is the hard case, and understanding why is the payoff of the streams model. TP splits a single layer's matmul across ranks: each rank computes a partial output, and an AllReduce sums the partials into the complete layer output before the next layer can run. Look at the dependency:

The next layer's matmul (Z) depends on Y, which depends on the AllReduce, which depends on this layer's matmul. The collective sits on the critical path — it is sandwiched between two compute steps that both need its result, and there is no other work on the compute stream to overlap it with. This violates the consumer-later condition by construction: the consumer is the very next kernel. TP comm is the most exposed communication in the whole series.

That single fact explains a rule you've already met: TP stays inside one node. Since the AllReduce can't hide, its full cost is exposed on every layer, twice per layer in the forward (attention + MLP) and again in backward — a very high frequency. The only lever left is to make each exposed AllReduce as cheap as possible, which means running it over the fastest fabric you have: intra-node NVLink (~900 GB/s peak, ~150 GB/s effective ring) rather than inter-node IB (~50 GB/s per NIC). You minimize the unhideable cost because you can't hide it.

The modern fix doesn't make the AllReduce hide behind a different layer — it can't — but behind itself. With sequence parallelism, the TP AllReduce is decomposed into its two halves (recall lesson 02's identity, AllReduce = AllGather ∘ ReduceScatter) spread across the layer boundary. Async tensor parallelism (Megatron's async-TP) goes further: it tiles the matmul and the collective together, so as each tile of the output matrix is computed, its partial result is communicated while the next tile is still being computed. The collective is now overlapped with the same matmul that feeds it, by interleaving at the tile granularity. It is fiddly, requires the matmul and the communication to be co-scheduled, and only pays off when the matmul is large enough to have tiles worth overlapping — but it is how frontier TP claws back some of the exposed time. The honest summary: TP comm is exposed by default, and the fixes are partial.

Case study — PP: tiny sends, the bubble is the cost (recall lesson 07)

Pipeline parallelism is the opposite extreme. Its communication is point-to-point Send/Recv of a single activation tensor across a stage boundary — small, infrequent, and trivially overlapped with the abundant compute of a pipeline stage. The Send of microbatch m's activations overlaps the compute of microbatch m+1 without any special effort. PP's communication is essentially never the exposed cost.

PP's exposed time is the bubble: the idle gaps at pipeline fill and drain where some stages have no microbatch to work on. And here's the connection that ties the lesson together — the famous PP schedules are overlap strategies in disguise. 1F1B (one-forward-one-backward) interleaves forward and backward microbatches so a stage always has work, shrinking the bubble. Zero-bubble schedules go further, splitting the backward into its input-gradient and weight-gradient halves and reordering them to fill the gaps almost completely. They don't reduce communication — they eliminate exposed idle, which is the pipeline's version of exposed time. Same principle, different resource: instead of hiding comm behind compute, you're hiding one stage's idle behind another stage's compute.

Interactive · the overlap timeline

Two stream lanes — compute (blue) and comm (orange) — for a single step. Slide the compute time, the comm time, and the overlap fraction (how much of the comm runs concurrently with compute). The hidden portion of the comm is drawn under the compute lane; the exposed remainder spills past it in red. Watch the effective step time collapse toward max(T_compute, T_comm) as overlap → 100%, and note that once you're compute-bound, dragging comm down does nothing — it's already free.

The takeaway the slider teaches: speedup from overlap is bounded by (T_compute + T_comm) / max(T_compute, T_comm). When the two are balanced, perfect overlap nearly halves the step. When one dominates, overlap can only erase the smaller one — there's nothing more to win. This is why people tune batch size and parallelism degrees to keep compute and comm in the same ballpark: that's where overlap has the most to give.

Interactive · the DDP bucket U-curve

This animates the DDP case study and sweeps the tradeoff. Backward runs left-to-right over L layers on the compute lane; as gradients fill each bucket, an AllReduce fires on the comm lane. Watch the last bucket — it fires after backward ends, and its red exposed tail is the irreducible DDP cost. The inset curve sweeps bucket size against total exposed comm: too-small buckets pay α latency on every one; too-big buckets leave a long exposed tail. The valley is the sweet spot.

Drag the bucket slider slowly from left to right while watching the inset dot. At the far left (tiny buckets) total exposed comm is high — dozens of AllReduces, each paying α, and the comm stream can't keep up with the gradients pouring in, so it backs up past the end of backward. At the far right (one giant bucket) it's high again — the single AllReduce only fires after backward finishes, so almost all of it is exposed. Somewhere in the middle the curve bottoms out: buckets small enough to start early and overlap, large enough to amortize α. That valley moves with bandwidth, α, and layer count — which is exactly why bucket size is a knob and not a constant.

Where the exposed time concentrates

Step back and the four case studies fall into a clean hierarchy of how hideable each scheme's communication is:

Reading down the "hidden behind" column is the design lesson: schemes whose communication has a later consumer (DDP, FSDP, PP) hide almost everything; the scheme whose consumer is the next kernel (TP) hides almost nothing. When you compose them in 3D parallelism (lesson 12), you place each scheme where its exposed cost is cheapest: TP intra-node on NVLink (smallest unhideable cost), DDP/FSDP across the wider, slower fabric (where the AllReduce hides behind backward anyway), PP across the slowest links (only tiny sends cross them).

The connection to MFU

Exposed communication is not an abstraction — it is the gap between the FLOPs your GPUs could have done and the FLOPs they actually did. An H100 delivers ~1 PFLOP/s of bf16; while a GPU stalls on exposed comm, it does zero useful FLOPs. Model FLOPs Utilization (MFU, lesson 23) measures exactly this — achieved throughput divided by peak — and every millisecond of T_exposed is a millisecond of stall that drags MFU down. The DDP tail, the TP critical-path AllReduce, and the PP bubble are not separate problems; they are the three places exposed time hides, and they are what your MFU number is really reporting.