Checkpointing & fault tolerance — surviving the 1000-GPU run

At one GPU the worst case is you lose an afternoon. At ten thousand GPUs for three weeks, hardware failure isn't an edge case — it's the expected hourly event. The last system you have to design isn't a parallelism strategy; it's the one that lets the job die and come back having barely noticed.

Why failure is the common case, not the rare one

A single GPU is reliable — call its mean time between failures M_1, maybe years. But failures are roughly independent across devices, so a job spanning N GPUs fails when any of them does, and the rates add:

M_job ≈ M_1 / N

Plug in numbers. If one accelerator (plus its host, NIC, optics, PSU) faults on average every ~50,000 GPU-hours, then a 10,000-GPU job has an effective MTBF of ~5 hours. A three-week run will be interrupted dozens of times. This is not pessimism — published large-model training logs report failures every few hours at scale, dominated by GPU faults, ECC errors, NIC/optics flaps, and the occasional host kernel panic. The synchronous nature of the collectives in lesson 02 makes it worse: one dead rank stalls an AllReduce, and the entire job hangs, not just the failed node.

What is actually in a checkpoint

"Save the model" is necessary and badly insufficient. To resume bitwise-equivalently — as if the failure never happened — you must restore every piece of mutable training state:

The headline: a 70B checkpoint is ~980 GB (params + optimizer state — gradients are transient and not saved), and the optimizer state, not the weights, is most of it. Saving only the weights (the common "I'll just save the model" mistake) makes the checkpoint look 7× smaller and makes resumption wrong: you'd restart Adam's moments from zero, spiking the loss every time you resume.

Failures over a run — feel the rate

Set the cluster size and the per-GPU MTBF and watch the expected failure count over a multi-week run. The point of this widget is visceral: at small scale the timeline is empty; drag the GPU count up and it fills with interruptions.

The central tradeoff — how often to checkpoint

Checkpoint too rarely and each failure costs a lot of lost recompute. Checkpoint too often and you spend the whole run writing checkpoints. There is a clean optimum, and it's one of the most useful formulas in the whole series.

Let C = the cost (time) of writing one checkpoint, M = the job MTBF, and τ = the interval (compute time) between checkpoints. Two sources of waste, as fractions of useful time:

• Checkpoint overhead = C / τ — you pay C every τ of work. Falls as τ grows.
• Lost-work overhead = τ / 2M — a failure (every M) discards on average half an interval. Rises as τ grows.

waste(τ) = C/τ + τ/(2M) ⇒ minimised at τ* = √(2 C M)

This is the Young–Daly formula (1974/2006), and the derivation is two lines of calculus. The τ/(2M) term carries one assumption worth stating: failures arrive uniformly in time (a Poisson process, independent of where you are in the interval), so a failure lands on average halfway through the current interval — you lose, in expectation, half an interval of work, τ/2, once every M of uptime, a fraction τ/(2M). Differentiate the total and set it to zero:

d/dτ [ C/τ + τ/(2M) ] = −C/τ² + 1/(2M) = 0 ⇒ τ² = 2CM ⇒ τ* = √(2CM)

(The second derivative 2C/τ³ > 0 confirms it's a minimum, not a maximum.) The minimum waste is √(2C/M) — it gets worse as checkpoints get expensive (C↑) or failures get frequent (M↓). The whole game of the rest of this lesson is driving C down, because a smaller C lets you checkpoint more often and waste less time doing it.

Driving down C — three mechanisms

A naïve checkpoint is a synchronous, single-writer dump: training halts while one rank streams ~1 TB to storage. At even a generous 2 GB/s that's ~8 minutes of dead time per checkpoint. With the Young–Daly lens, a huge C forces huge intervals and large lost-work. Three ideas, stacked, take C from minutes to seconds.

1 · Sharded / distributed checkpoint

Under FSDP/ZeRO (lesson 05) each rank already holds only 1/N of the state. So let each rank write its own shard in parallel: aggregate write bandwidth scales with the number of writers, and wall-clock write time drops by N×. This is what torch.distributed.checkpoint (DCP) does. It also stores state logically (by parameter name + offset, not by physical rank layout), which enables the next trick.

2 · Asynchronous checkpoint

Split the write into a fast copy and a slow flush. Snapshot the state from HBM into pinned host memory — bounded by PCIe at ~25 GB/s, a few seconds — then resume training immediately while a background thread drains host memory to storage. The training stall shrinks from "time to write to disk" down to "time to copy to RAM." The disk write still happens; it just no longer blocks the GPUs.

3 · In-memory / redundant checkpoint

For the fastest recovery, keep a copy of each rank's shard in a peer GPU/host's memory (erasure-coded or replicated, à la Gemini / CheckFreq). When a node dies, its neighbour already holds its last shard — recovery is a memory copy over NVLink/IB, not a multi-minute read from a cold object store. Disk checkpoints remain as the durable fallback for whole-cluster outages.

Recovery — detection, restart, and elasticity

A fast checkpoint is half the story; you also have to come back. The recovery loop:

1. Detect. A health monitor (or a collective timeout) notices a rank is gone. Because collectives hang on a missing peer, you need an explicit watchdog — otherwise the job silently stalls forever instead of failing.
2. Reschedule. Either swap in a hot spare node, or — with elastic training (torchrun/TorchElastic) — re-form the process group with the surviving nodes at a smaller world size. Elasticity trades a little throughput for not needing the exact node count to be available.
3. Restore. Every rank loads the latest checkpoint (resharded to the new topology if the world size changed) and resumes from the saved step, LR position, RNG, and data offset.

Detecting the failure — the recovery loop starts here

All the write machinery above is dead weight if you can't tell something broke — and detection at thousand-GPU scale is the part everyone underestimates. The naïve mental model is "a process crashes, the OS notices, you restart." Reality is uglier: the common failure mode is not a clean crash but a hang. One rank dies or slows, and every other rank blocks indefinitely inside a collective. This is a direct consequence of lesson 02: a collective only returns when all ranks have called it. If rank 743's GPU faults mid-step, ranks 0–742 and 744–8191 all enter the next AllReduce, wait for a peer that will never arrive, and sit there — burning power, holding the allocation, making zero progress, and emitting no error. A run can be "alive" by every process-liveness check while being thoroughly, silently dead.

So you need explicit mechanisms that turn a silent hang into a loud, localized fault:

• Collective timeout. NCCL/PyTorch arm a watchdog on every collective (TORCH_NCCL_BLOCKING_WAIT / the async error-handling path). The default timeout is generous — on the order of 10–30 minutes — because legitimate collectives can be slow. But 30 minutes of a 10,000-GPU cluster idling on every failure is enormous wasted goodput, so you tune it down (to minutes) once you know your steps' real latency. Too low and a slow-but-healthy step trips a false abort; too high and you pay for the stall. It's a tuning knob, not a default to leave alone.
• The NCCL flight recorder. Setting TORCH_NCCL_TRACE_BUFFER_SIZE keeps a ring buffer of the last k collectives each rank issued — op type, sizes, sequence numbers, and which completed. When the watchdog fires it dumps these traces. The diagnostic gold is the mismatch: 8191 ranks have issued AllReduce #40,118 and are waiting; one rank's last recorded op is #40,117. That rank is your culprit — it stopped calling collectives, so it's where the GPU/host died. Without the flight recorder you know that the job hung; with it you know which node to power-cycle or evict.
• Heartbeat / watchdog process. A side process (per node, plus a job-level coordinator) emits a heartbeat on an interval. Miss n beats and it's declared dead, independent of whether its collectives are stuck. This catches the case where the GPU is wedged but the kernel and NIC are fine, so lower-level timeouts haven't fired.

The reason for all three is that you must distinguish three distinct fault classes, because they route to different responses:

Checkpoint durability — the checkpoint itself can be poison

Here is the failure mode that turns your safety net into the thing that kills the run: a crash during a checkpoint write. Failures are routine (M_job = M_1/N), and a checkpoint write takes seconds-to-minutes, so eventually a node will die while a checkpoint is being written. The result is a torn checkpoint — a file (or a shard) that is half-old, half-new, or truncated. If the canonical "latest" name now points at that partial blob, the next restart loads garbage, and either crashes immediately (best case) or silently corrupts the run by restoring inconsistent state across shards (worst case). The thing you built to survive failures has become the failure.

The defenses are cheap and non-negotiable:

• Atomic publish via temp-path + rename. Write to a scratch path (step_40000.tmp/), fsync it, and only then os.rename it to the canonical name (or flip a latest pointer/symlink). rename within a filesystem is atomic, so the canonical name only ever resolves to a fully-written checkpoint. A crash mid-write leaves a stray .tmp you garbage-collect on restart — it never poisons the pointer.
• Keep the last N checkpoints; delete late. Do not delete checkpoint k-1 until checkpoint k has been written and verified. Retaining a small ring (say 3–5) costs storage but means a single torn or unreadable write never leaves you with zero good checkpoints — you fall back one step.
• Checksum / verify-load before trusting. Store a checksum per shard and validate on load; better, in async mode (where a background thread is doing the flush anyway), do a trial deserialize of the just-written checkpoint and only advance the latest pointer once it loads clean. A checkpoint you've never successfully read is a hope, not a backup.
• History buys rollback depth. The newest checkpoint can be valid-on-disk yet semantically poisoned — e.g., it captured a NaN (next section). Keeping history lets you roll back past the most recent two or three checkpoints to the last point the math was healthy, not just the last point the bytes were intact.

When the math dies — NaN as a fault

Close the loop with lesson 24: a loss spike or a NaN is a failure too — and the powerful realization is that it reuses the exact same machinery as a dead GPU. There are two ways a training step can stop being useful: "a GPU died" (hardware) and "the math died" (a NaN/Inf propagated into the weights). Both route into one path: detect, roll back to the last good checkpoint, resume. You do not need a separate recovery system for numerical failure; you need to feed it into the one you already built.

• Detect collectively. Each rank computes a local isfinite flag over its loss/gradients and you all-reduce it (a max, or a sum) across all ranks. The single resulting bit answers "did any rank see a NaN this step?" — which matters because a NaN on one shard will, after the next AllReduce of gradients, contaminate every rank's weights. One global flag, one branch.
• Roll back. If the flag is set, discard the current (now-poisoned) state and reload the last known-good checkpoint — which is exactly why durability and history (previous section) matter: the checkpoint written 30 seconds ago may already contain the NaN, so you may need to step back two or three.
• Skip and/or de-risk before resuming. Often a specific data shard triggered the spike (a pathological batch). Using lesson 25's resumable data offset, you can resume past the offending shard so you don't immediately re-trigger the same NaN, and/or lower the LR (or re-warm it) for a few hundred steps to ride through the unstable region before restoring full LR.

Goodput — optimize the whole loop, not just the write

Young–Daly's cost term C is seductively narrow: it's easy to read it as "checkpoint write time" and declare victory once async flush has driven the stall to a few seconds. But a real failure costs far more than the write. The end-to-end recovery cost is:

C_recover = detect + reschedule + reload + recompute_since_last_checkpoint

Each term is real and independently large. Detect is the timeout/heartbeat latency from the first section — if your collective timeout is 20 minutes, every failure burns 20 minutes of the whole cluster before recovery even begins. Reschedule is getting a replacement node — instant if you have a warm hot spare, minutes-to-tens-of-minutes if the scheduler has to find and boot one. Reload is reading the (now cold) checkpoint back. Recompute is the Young–Daly lost work, ~τ/2. The write being cheap does nothing for the first three.

The metric that captures all of it is ML goodput:

goodput = useful_training_wall_clock / total_wall_clock

Useful wall-clock is time spent doing forward/backward passes that survive to the final model; everything else — checkpoint stalls, detection latency, rescheduling, reloads, and recomputed-then-discarded work — is in the denominator but not the numerator. The trap: you can have a near-zero checkpoint write cost and still bleed goodput, because frequent small failures with slow detection dominate. At a 5-hour job MTBF with a 20-minute detect-and-reschedule per failure, a three-week run eating ~100 failures loses ~33 hours to detection alone — independent of how fast the write is. Optimizing C's write term while ignoring detect/reschedule is optimizing the wrong term.

This reframes the engineering targets:

• Hot-spare pool sizing. Run with N + s nodes where s idle spares stand ready. Sizing s is a goodput calculation against lesson 01's M_job = M_1/N: too few spares and recovery stalls waiting for the scheduler; too many and you're paying for idle GPUs. You size s so the expected concurrent-failure count rarely exceeds it.
• Fast re-formation. When a node dies, the job has to re-form. The scheduler (SLURM / Kubernetes) requeues or swaps the node, and torchrun/elastic uses a rendezvous backend (c10d/etcd) to re-establish the process group — surviving ranks plus the replacement re-handshake into a fresh world. If a spare isn't available, elastic re-forms at a smaller world size, which (as the lesson's elastic-restart note covers) shrinks N and forces a re-solve of B_eff / accumulation G so the effective batch — and thus the LR schedule — stays pinned across the reshape.
• Detection is a goodput lever, not just a safety feature. Cutting the collective timeout from 20 minutes to 3 directly multiplies surviving wall-clock at high failure rates. The flight recorder pays for itself the same way: fast localization means you evict the right node on the first try instead of restart-looping.

The whole picture

  ───●────────────────●────────────────✗  failure (rank dies)
     │ checkpoint      │ checkpoint      │
     └── τ work ───────┴── τ work ───────┘  lost: ~τ/2 of recompute
                                          │
                          detect (watchdog) → reschedule (spare / elastic)
                                          │       → restore latest checkpoint
                                          ▼          (reshard if world size changed)
  ───────────────────────────────────────●──────▶  resume at saved step

   τ* = √(2·C·M)        drive C down with: sharded write · async flush · in-mem redundancy

Synthesis — MFU × goodput, the two halves of efficiency

Part VI closes by putting two numbers in one equation that the series has so far kept apart. MFU (lesson 23) measures compute efficiency within useful wall-clock: of the seconds you spend doing forward/backward that survives to the final model, what fraction of the FLOP-budget went to useful math. Goodput (this lesson) measures what fraction of total wall-clock was useful at all — the rest lost to crashes, restarts, detection latency, reloads, and checkpoint stalls. They are the same efficiency family, decomposed multiplicatively: one is efficiency given the clock is useful, the other is the chance the clock is useful in the first place.

effective_tokens_per_GPU_per_dollar ∝ MFU × goodput

The product is what you actually pay for. A run at a stellar 55% MFU but 60% goodput — great kernels, but it crashes every few hours and each recovery bleeds detection-and-reload time — delivers the same useful throughput as a mediocre 38% MFU run at 87% goodput, and far less than a 50% × 95% run. A high-MFU run with poor goodput still wastes the cluster: the tensor cores are magnificently efficient during the shrinking slices of wall-clock that survive, and idle or recomputing-then-discarding during the rest. You cannot optimize one and ignore the other — MFU × goodput is the single quantity that says whether the weeks of GPU-time you rented turned into a model. That product, tying the compute story of Part V and the operational story of Part VI together, is the real scoreboard for a large-scale run.