Part II — Systems

Running the job — surviving a multi-week run

Lessons 06–10 handed you a training machine that fits and runs fast: FlashAttention keeps the GPU fed, FSDP and tensor/pipeline parallelism let any model fit, MoE buys parameters off the cheap axis. On a laptop-scale job you would now just press run and watch the loss fall. But a frontier run is not a laptop job. It is three weeks of wall-clock on a thousand-plus GPUs, and over that span the hardware will fail, the loss will occasionally spike, and a process that simply restarts from step 0 — or, worse, restarts from a checkpoint that doesn't quite match — sets fire to the budget you spent two lessons learning to protect. Executing the run is its own engineering discipline: the job has to be a fault-tolerant, deterministic, resumable system, not a script.

The previous step left this broken

The systems half (06–10) optimized one thing: the throughput of a healthy step. It assumed the run runs. It does not — not for weeks, not at this width. A single H100 has a finite mean time between failures; multiply by a thousand of them held in lockstep by collectives (every all-reduce in lesson 8, every pipeline bubble in lesson 9) and a single dead GPU stalls or crashes all of them. Add the loss spikes lesson 4 warned about, plus host crashes, NCCL hangs, ECC errors, and a flaky network link, and the question is no longer "is each step fast?" but "how much of the budget survives three weeks of things breaking?" Nothing in 06–10 priced a crash. The accounting in lesson 5 assumed every GPU-hour you rent turns into training; in reality a chunk of it is recompute after failures, and an undisciplined run lets that chunk balloon.

Linear position

Forced by: 06–10 give a machine that fits and runs fast, but a frontier run is weeks long on thousands of GPUs where hardware will fail and the loss will spike — and a run that can't resume bit-for-bit, or can't restart itself, burns the budget you spent learning to protect.
New idea: treat the training job as a fault-tolerant, deterministic, resumable system — correct checkpoints (every byte of state, not just weights), deterministic sharded data loading, automatic detect-and-restart at scale, an operationalized spike runbook, and the dashboard you actually watch. Wasted GPU-hours, not peak FLOPs, is the metric.
Forces next: you can now run a job and finish it in budget — but you still have not decided how big the model should be, and you get exactly one shot: you cannot afford to run it twice to find out. That forces scaling laws, lesson 12.

The plan

Six moves. (1) The wall: a crash converts GPU-hours straight into burned budget, and at scale it happens repeatedly. (2) The checkpoint: the exact state you must save to resume bit-for-bit — params, optimizer moments, the schedule step, RNG, the data cursor — sized by the lesson-5 ledger, written asynchronously so saving doesn't stall the run. (3) Deterministic sharded data loading so G ranks see disjoint, reproducible, resumable data. (4) Fault tolerance at scale: an MTBF argument that says you will get ~10 failures in three weeks, so you must auto-restart. (5) Spike recovery as a runbook (callback lesson 4). (6) The monitoring dashboard and what each signal means. Then drive the run-survival simulator until "never checkpoint" detonates.

1 · The wall: a crash is budget, not an inconvenience

Recall the currency from lesson 0: the budget is denominated in GPU-hours, and lesson 5's runtime estimate, time ≈ 6ND / (GPUs · peakFLOPs · MFU), assumed every rented GPU-hour becomes forward progress. A crash breaks that assumption directly. When a process dies, every GPU rolls back to the last checkpoint and recomputes everything since — and that recompute is GPU-hours you already paid for, spent buying back progress you already had.

Put numbers on it. You are 200 hours into a run on 1,024 H100s, and your last checkpoint was written 6 hours ago. One GPU throws an uncorrectable ECC error and the job dies. To resume you reload the 6-hour-old checkpoint and redo those 6 hours — on all 1,024 GPUs:

lost ≈ 6 h × 1,024 GPUs = 6,144 GPU-hours — gone, for one crash

At a representative cloud rate of a few dollars per H100-hour, that single failure is tens of thousands of dollars of recompute, and it bought you nothing — you are back where you were six hours ago. Now recall this is not a one-off: section 4 shows a three-week run on this cluster expects roughly ten such failures. Ten times 6,144 is over 60,000 GPU-hours of pure recompute — a real dent in a multi-hundred-thousand GPU-hour budget — and that is the good case, where every crash is detected and restarted promptly. The undisciplined case (rare checkpoints, manual restart, a stale checkpoint that forces you to redo even more) is far worse. The job of this lesson is to shrink that wasted fraction toward zero.

The lever you actually control

You cannot make hardware stop failing. The two things you can control are how much work each crash costs (checkpoint frequency — the recompute per failure is, on average, half the checkpoint interval × the GPU count) and how fast you detect-and-restart (automation — minutes of idle cluster per crash, not the hours a sleeping human costs). Everything below is one of those two levers.

2 · The checkpoint: what must be saved to resume bit-for-bit

A checkpoint's job is to let a resumed run continue as if the crash never happened — the loss curve after resume should lie exactly on the curve you would have gotten without the crash. That is a strict requirement, and it is where the naive "just save the weights" instinct fails. Saving only model parameters produces a run that limps after resume: the loss jumps, then slowly re-converges, silently wasting hundreds of steps. The reason is that the optimizer and the schedule carry hidden state that the weights alone don't capture. To resume bit-for-bit you must persist all five of these:

model parameters

The weights themselves (fp32 master copy — lesson 4). Obvious, and the only thing most people remember to save.

optimizer moments m & v

AdamW's first and second moments (lesson 4). Drop these and the optimizer restarts cold: its per-coordinate step sizes are wrong for thousands of steps. This is the single most common cause of a post-resume loss bump.

LR-schedule step counter

The global step index t. The warmup→cosine schedule (lesson 4) is a function of t; resume with t = 0 and you re-warm-up mid-run, spiking the LR and the loss.

RNG states

The PRNG state on every rank — for dropout, any stochastic layers, and (critically) the data shuffle. Miss it and the post-resume randomness diverges from the original trajectory.

data-loader cursor

Exactly how far each rank has read into its data shard (§3). Without it, resume either re-reads data already seen (extra epochs on a slice) or skips ahead — either way the data distribution the model trained on no longer matches the intended one.

Miss any one and the resumed run is no longer the same run. The discipline is blunt: a checkpoint is the entire training state, and "the model" is only the first of five parts.

How big is a checkpoint? The lesson-5 ledger, again

A checkpoint is dominated by the optimizer/master state, so its size is the 16-bytes-per-parameter ledger from lesson 5: fp32 master weight (4) + Adam m (4) + Adam v (4) — that is the 12 bytes you must persist — plus the working bf16 copies. For a 7B model:

7×10⁹ params × 16 B/param ≈ 112 GB per checkpoint (the optimizer+master+params state of lesson 5)

That is per checkpoint, and you write many of them. The frequency is a direct trade-off, and it is exactly the knob the widget exposes:

Checkpoint every…	Expected recompute per crash	Cost of checkpointing
too often (e.g. 5 min)	tiny — lose ≤ ~2.5 min × GPUs	high — constant I/O stalls / overhead eats throughput
sweet spot (~30–60 min)	small — lose ~½ interval × GPUs	low — a few % of one step's time, amortized
too rarely (e.g. 6 h)	large — §1's 6,144 GPU-hours per crash	negligible — but the recompute dominates

There is a sweet spot because the two costs pull opposite ways: shrinking the interval cuts recompute-per-crash linearly but raises the number of (costly) saves, so total waste is a U-shaped curve with an interior minimum. The widget plots exactly this.

Why checkpointing must be asynchronous and sharded

The naive checkpoint stalls the run: every GPU stops, the full state is copied to host and streamed to durable storage, and only then does training resume. At 112 GB (and far more for a frontier model) that stall is minutes — and you pay it on every save, on every GPU. Two techniques make the stall nearly free:

sharded (distributed) checkpointEach FSDP rank (lesson 8) writes only the parameter/optimizer shard it already holds, in parallel, to its own file. No rank gathers the full model; aggregate write bandwidth scales with the GPU count instead of bottlenecking on one rank.

asynchronous writeSnapshot the state to pinned host memory fast (a device-to-host copy), then let a background thread stream that snapshot to durable storage while training continues. The GPUs stall only for the quick copy, not the slow upload.

in-memory / redundant copiesFor the fastest restart, also keep a recent checkpoint replicated in the cluster's RAM (on peer nodes), so a single-node failure restores from memory in seconds rather than re-reading the durable store.

The result: the GPU-visible cost of a checkpoint drops from minutes to a handful of seconds, so you can afford to checkpoint often — which is what pushes the recompute-per-crash term toward zero. It does not vanish entirely, though: each save still costs that brief snapshot-and-barrier, so checkpointing too often spends real throughput on overhead. That residual cost is what gives the wasted-hours curve its sweet spot rather than a free slide to zero — exactly the U the widget plots.

3 · Deterministic, sharded data loading

The data loader is where the most insidious bugs live, because they corrupt the training distribution silently — the loss looks fine, the model is quietly worse. The setup: with G data-parallel ranks (lesson 8), each step the global batch is split across ranks, so each rank must stream a disjoint shard of the corpus. Two properties are non-negotiable.

disjoint, balanced shards

Rank r of G reads a partition no other rank touches (e.g. interleave by index r, r+G, r+2G, …, or hash-assign documents to ranks). Every training example is seen by exactly one rank per epoch.

a global seed → reproducible order

One seed determines the shuffle for the whole run. The order in which examples appear is then a deterministic function of (seed, step) — re-runnable, and crucially, recoverable on resume.

Resume then becomes well-defined: the run records the data-loader cursor (§2) in the checkpoint, and on restart the loader fast-forwards to exactly the next unseen sample of each rank's shard — the very next batch is the one that would have come next had the crash never happened. The classic bugs are both failures of this contract:

Bug A — overlapping shards

Two ranks compute their shard from the same seed but forget to offset by rank, so they stream the same examples.
The gradients those ranks contribute are duplicates, so the all-reduced gradient is biased toward the duplicated data — effectively a smaller, skewed batch.
Signal: throughput looks normal, but the model behaves as if trained on less (or oddly weighted) data; the effective batch is smaller than you think.

Bug B — reshuffle on resume

The shuffle is re-seeded from wall-clock time (or the cursor isn't saved), so after a restart the order is different from the original.
Some examples get seen twice (extra, unintended epochs on a slice → memorization) and others get skipped entirely that epoch.
Signal: a small loss discontinuity at every resume; over a run with ten restarts, a measurable chunk of data seen the wrong number of times.

Both are invisible on the loss curve in the short term and corrosive over a long run. The fix is the same discipline as the checkpoint: the data order is a pure, seeded function of the step, and the cursor is part of the saved state.

4 · Fault tolerance at scale: you will get ~10 failures

Here is the argument that forces automation. Suppose a single accelerator (plus its host, network link, power) has a mean time between failures of roughly 50,000 GPU-hours — optimistic for a healthy fleet. Failures across independent GPUs add: the fleet's expected failures over a run is just total GPU-hours divided by the per-GPU MTBF.

expected failures ≈ (GPUs × hours) / MTBF

Run the numbers on 1,024 GPUs. The fleet burns 1,024 GPU-hours every wall-clock hour, so it accumulates 50,000 GPU-hours — one expected failure — in about 50,000 / 1,024 ≈ 49 hours, i.e. roughly every two days. A three-week (21-day) run therefore expects:

(1,024 × 21 × 24) / 50,000 = 516,096 / 50,000 ≈ 10.3 failures

Ten interruptions in three weeks, each one freezing all 1,024 GPUs until something acts. If "something" is a human paged at 3 a.m., each failure costs the time-to-notice plus time-to-restart — easily an hour of a fully idle cluster (1,024 idle GPU-hours) on top of the recompute. That is the "babysitting the run" reality teams describe, and it does not scale: at 10,000 GPUs you are failing every few hours, around the clock. The interruptions must be handled automatically:

detect

A watchdog notices a dead rank — a crashed process, an NCCL collective that hangs past a timeout, a host that stops heartbeating, an ECC/Xid error in the logs.

restart elastically

Elastic training (e.g. torchrun elastic): the launcher tears down the broken process group and re-forms it on the survivors (or after swapping in a spare), every rank reloads the last checkpoint, and the run continues — no human in the loop.

hot spares

Keep a few idle nodes provisioned so a failed node is replaced, not just dropped — preserving the parallelism layout (the DP×TP×PP mesh of lesson 9 often needs its shape back).

This is precisely what cluster orchestration layers are built to provide. The deeper machinery — how a scheduler detects dead workers, reconstitutes a process group, and reschedules onto spares — is treated in the Ray track (distributed execution and fault-tolerant actors) and in the reinforcement-learning track's distributed-topology lessons, where the same detect-and-restart loop wraps a far more dynamic (rollout + train) workload. The principle is identical: a long job on many machines must assume machines die and recover without you.

5 · Spike recovery, operationalized (the lesson-4 runbook)

Lesson 4 taught why loss spikes happen (heavy-tailed gradients, a pathological batch, too-hot an LR) and the levers that prevent most of them (warmup, global-norm clipping at 1.0, a sane peak LR, β₂ = 0.95). But on a frontier run, even with all four levers on, a spike eventually gets through — a genuinely bad document, a rare numerical event. Prevention is lesson 4; this is the runbook for when it happens anyway, in increasing order of cost:

1 · detectThe monitor (next section) flags a loss jump or a gradient-norm spike that clipping didn't fully absorb. Catch it early — a spike heading toward NaN is unrecoverable once weights go inf.

2 · rewindRoll back to the last good checkpoint — the last one written before the spike. This is why frequent checkpoints matter: you lose only the work since that checkpoint, not the run.

3 · skip the offending batchesA specific span of data often causes the spike. Resume from the good checkpoint but advance the data cursor (§3) past the offending batches so the optimizer never sees them again.

4 · soften, if it recursIf skipping isn't enough, lower the peak LR a notch and/or tighten gradient clipping (and optionally lower β₂), then resume. A small, deliberate stability tax beats another spike.

Every step here depends on §2 and §3 being correct: rewinding needs a recent, bit-for-bit checkpoint, and skipping batches needs a deterministic, addressable data cursor. The spike runbook is not a separate system — it is what a correctly resumable run lets you do. A run that can't resume cleanly can't execute this playbook, and a spike that would have cost an hour costs the run.

6 · Monitoring: the dashboard you actually watch

You cannot fix what you cannot see, and a run is mostly invisible — a thousand GPUs grinding in a datacenter. The instrument is a live dashboard, and four signals carry almost all the information. Each one tells you about a different failure class from the lessons above.

loss

The primary instrument (lesson 4 §8). Smooth descent = healthy; a vertical jump = a spike (run the §5 runbook); a flat line or NaN = diverged or dead. The first thing your eyes go to.

gradient norm

‖g‖ before clipping. A sudden upward spike is the earliest warning of instability — it moves before the loss does, so it's your leading indicator. Persistent growth means the run is going unstable.

throughput / MFU

Tokens/s and achieved-vs-peak FLOPs (lesson 5–6). A sudden drop means a straggler GPU, a degraded network link, or comm not overlapping — money leaking even though nothing has crashed.

hardware health

Per-GPU temperature, power, ECC error counts, Xid errors, link bandwidth. A climbing ECC count predicts the crash from §1 before it happens — so you can checkpoint and pre-empt it.

Read together they localize a problem fast: loss spike with a gradient-norm spike → data/optimizer event (runbook §5); MFU drop with one GPU's health degrading → straggler, drain that node; loss flat and a rank stopped heartbeating → a crash, confirm the auto-restart (from §4) fired. The dashboard is how the human supervises a system that is mostly automated — watching the signals that automation can't yet act on, and confirming that it acted on the ones it can.

7 · Drive the run-survival simulator

The widget turns this whole lesson into one number you care about — GPU-hours wasted — and shows the checkpoint-frequency sweet spot directly. Set the cluster size, the per-GPU MTBF, the checkpoint interval, and the run length. It computes (deterministically — these are expectations, not a dice roll) the expected failures from §4, the recompute lost per crash from §1, and the checkpointing overhead from §2, and reports the effective MFU you keep. Two things to make happen:

Find the U. At 1,024 GPUs and a 21-day run, sweep the checkpoint interval from 5 min upward. Very short intervals waste GPU-hours on checkpointing overhead; very long intervals waste them on recompute per crash. The wasted-hours readout bottoms out in the middle — the sweet spot the §2 table described.
Break it. Drag the checkpoint interval all the way to "never". Now every crash rewinds to step 0, so each of the ~10 expected failures costs half the run on average — wasted hours explode and effective MFU collapses. That is the catastrophe the entire lesson exists to prevent.

Run-survival simulator — wasted GPU-hours vs the checkpoint sweet spot

Expected failures ≈ (GPUs × days × 24) / MTBF (§4). Recompute per crash ≈ ½ × checkpoint-interval × GPUs (§1). Overhead ≈ (#checkpoints) × (a few-second sharded-async stall) × GPUs (§2). Shrink the interval and overhead climbs; grow it and recompute climbs — wasted hours are U-shaped with an interior sweet spot near ~30 min. Push the interval to "never" at 1,024 GPUs and each crash rewinds to step 0 → catastrophic loss. All values are deterministic expectations (no randomness).

# GPUs: 1024 per-GPU MTBF (h): 50000 checkpoint interval: 60 min run length (days): 21

useful training recompute after crashes checkpointing overhead

Expected failures

—

GPU-hours wasted

—

Effective MFU kept

—

Lost per crash

—

Checkpoint overhead

—

Verdict

—

The reading to internalize: wasted hours = recompute + overhead, and the only free lunch is making checkpoints cheap (the sharded-async trick of §2) so you can afford to take them often, pushing the sweet spot left toward zero recompute without the overhead arm rising to meet it. Everything else is a trade. And note what the widget does not let you do: set MTBF to infinity. Hardware fails; the question is only how much of the budget you let each failure take.

Failure modes & checklist

Failure modes

Checkpointing only the weights. Optimizer moments, schedule step, RNG, and data cursor are dropped. Signal: a loss bump every time you resume, then slow re-convergence — hundreds of wasted steps per restart.
Checkpoint interval too long. Saving rarely "to avoid overhead." Signal: each crash costs many GPU-hours (§1); cumulative recompute is a large fraction of the run.
Synchronous, unsharded checkpoints. The whole cluster stalls for minutes on every save. Signal: periodic throughput dips exactly at the checkpoint cadence; you respond by checkpointing less — straight into the previous failure.
Overlapping or reshuffling data shards. Ranks see duplicate data, or the order changes on resume. Signal: effective batch smaller than intended, or a small loss discontinuity at every restart; quietly worse model.
No automatic restart. A human babysits the run. Signal: hours of fully idle cluster per failure; with ~10 failures over three weeks, the idle time alone is thousands of GPU-hours.
No leading indicators on the dashboard. Watching only the loss. Signal: you catch spikes late (loss lags gradient norm) and crashes late (after the collective times out, not when ECC errors started climbing).

Checklist

Save all five state pieces — params, optimizer m/v, schedule step, RNG (every rank), data cursor — and verify by resuming and confirming the loss curve is continuous.
Checkpoint asynchronously and sharded so a save costs ~seconds, then checkpoint often (≈ every 15–60 min) to shrink recompute-per-crash.
Make data order a pure seeded function of the step; shard disjointly by rank; persist and fast-forward the cursor on resume.
Run elastically with hot spares — auto-detect dead ranks, re-form the group, reload, continue; no 3 a.m. page.
Keep the spike runbook ready — rewind to last good checkpoint, skip the offending batches, soften LR/clip if it recurs (lesson 4).
Watch loss, gradient norm, MFU, and hardware health together; treat a climbing ECC count as a checkpoint-now signal.

Checkpoint

Try it

Open the widget at the defaults (1,024 GPUs, MTBF 50,000 h, 60-min checkpoints, 21 days). Confirm the expected-failures readout is ≈10.3 (the §4 number) and note the GPU-hours wasted and effective MFU. (a) Sweep the checkpoint interval from 5 min up to 360 min and find the value that minimizes wasted hours — that is the sweet spot; note how the wasted total rises on both sides of it. (b) From the sweet spot, drag the interval to "never" and read the new wasted hours and effective MFU — explain in one sentence why one design choice can turn a ~2% tax into a budget-ending one. (c) Raise the cluster to 8,192 GPUs at the same interval: confirm expected failures scale ≈8× and that you now want to checkpoint more often. Finally, by hand: at 1,024 GPUs with the last checkpoint 6 hours ago, reproduce §1's 6,144 GPU-hours-per-crash figure, and say which of the two levers (frequency vs auto-restart) that number is about.

Where this points next

You can now execute a frontier run: it survives dead GPUs because it auto-restarts and resumes bit-for-bit from cheap, frequent, complete checkpoints; it survives loss spikes because the runbook can rewind and skip; and you can watch it stay healthy on a dashboard whose signals you can read. The wasted-GPU-hours fraction is small and bounded, so the lesson-5 runtime estimate is now honest — the budget you planned is roughly the budget you spend. But that only sharpens the question this whole part has been deferring. You can run a job, end to end, without burning the budget — yet nothing has told you how big the model should be: how to split your fixed compute between parameters N and tokens D, with the MoE knobs from lesson 10 layered on top. And the cruelty established here makes it acute: a frontier run is millions of GPU-hours and you get one attempt — you cannot run it twice to see which size was right. So you must predict the best configuration before you spend a FLOP, from cheap small-scale experiments. That is the entire subject of the next part. Next: 12 · Scaling laws — Chinchilla-optimal.

Takeaway

A frontier run is weeks of wall-clock on thousands of GPUs, and over that span hardware will fail and the loss will occasionally spike — so the systems half's fast, fitting machine (06–10) is not enough; the run itself must be a fault-tolerant, deterministic, resumable system, or it burns the very budget you learned to protect. A crash converts GPU-hours straight into recompute: a 6-hour-old checkpoint on 1,024 GPUs costs 6 × 1,024 = 6,144 GPU-hours per crash, and a 21-day run on that cluster expects ≈(1,024 × 21 × 24)/50,000 ≈ 10 crashes. You attack this with two levers: cheap, frequent, complete checkpoints — all five of params, optimizer m/v, the schedule step, RNG, and the data cursor (miss one and resume isn't bit-for-bit), sized by the lesson-5 16-B/param ledger (≈112 GB for 7B), written sharded and asynchronously so a save costs seconds — and automatic detect-and-restart (elastic training + hot spares) so no human babysits the cluster. Deterministic, disjoint, seeded data shards with a saved cursor keep resume from duplicating or skipping data; the lesson-4 spike runbook (rewind → skip batches → soften LR/clip) is what a resumable run lets you do; and a dashboard of loss, gradient norm, MFU, and hardware health is how you supervise it, with gradient norm and ECC counts as the leading indicators. Done right, the wasted fraction is a couple of percent and the lesson-5 runtime estimate is honest. But you still haven't chosen N and D, and you get one shot — so you must predict the right size before spending it: scaling laws, lesson 12.

Interview prompts

Exactly what state must a checkpoint contain to resume bit-for-bit, and what breaks if you save only the weights? (§2 — model params, AdamW moments m & v, the LR-schedule step counter, RNG state on every rank, and the data-loader cursor. Save only weights and the optimizer restarts cold, the schedule re-warms-up, and the data order desyncs — the loss bumps on resume and re-converges slowly instead of continuing on the original curve.)
Estimate the budget a single crash costs and how often crashes happen at scale. (§1, §4 — recompute ≈ time-since-checkpoint × GPUs: a 6-hour-old checkpoint on 1,024 GPUs is 6,144 GPU-hours per crash. Expected failures ≈ GPUs×hours/MTBF; at MTBF≈50,000 h a 21-day 1,024-GPU run expects ≈10 failures — one roughly every two days.)
How should checkpoint frequency be chosen, and why is there a sweet spot? (§2 — recompute-per-crash ≈ ½ × interval × GPUs falls as the interval shrinks, but the number of (costly) saves rises, so total wasted hours are U-shaped with an interior minimum. Making saves cheap — sharded + async — pushes the optimum toward frequent checkpoints and near-zero recompute.)
Why must checkpointing be asynchronous and sharded, and how does each part help? (§2 — sharded: each FSDP rank writes only the shard it holds, in parallel, so write bandwidth scales with GPU count instead of bottlenecking one rank. Async: snapshot to pinned host memory fast, then stream to durable storage in the background while training continues — the GPU stall drops from minutes to ~a second.)
Name the two classic data-loader bugs in a sharded run and how each corrupts training. (§3 — overlapping shards: ranks stream the same examples, so all-reduced gradients are duplicated/biased and the effective batch is smaller and skewed. Reshuffle-on-resume: the order changes after a restart, so some data is seen twice (extra epochs → memorization) and some skipped. Fix: data order is a pure seeded function of the step; persist and fast-forward the cursor.)
What is the operational runbook when the loss spikes mid-run, and what does it depend on? (§5 — detect early (gradient norm leads loss), rewind to the last good checkpoint, skip the offending data batches via the cursor, and if it recurs lower peak LR / tighten clip (lesson 4). It depends entirely on §2–§3: a recent bit-for-bit checkpoint and a deterministic, addressable data cursor.)
Which four signals do you watch on the run dashboard, and what failure does each catch? (§6 — loss (spikes/divergence, but lagging), gradient norm (the leading indicator of instability, moves before loss), throughput/MFU (stragglers, degraded links, lost overlap — money leaking with no crash), and hardware health/ECC (predicts the crash before it happens, so you checkpoint and pre-empt).)