The exam we never stopped taking: evaluation

Lesson 15 ended with a decoy: a better one-step score, a worse closed loop. This lesson makes the decoy a census. Twelve one-jump models of the Courtyard, differing in capacity, data, the nudges in their logs and a hidden confounder, sit five cheap exams and the decision they exist for; Kendall's τ says which exam ranks them as the decision does. None does for every use: the held-out error is best at the series' goal and worse than chance at a moved one, where the exam that tests the whole action range is best. The decision can be measured, at a price we derive: how many launches, what zero failures prove, how far a winner is flattered. Ten failures from lessons 3 to 15 each have an exam that passes them and one that catches them. The lesson says how good a model is, not how to make it better.

1 · Twelve models of one world, and the exam that matters

The task is lesson 1's nudge: one impulse of at most 3 m/s at step 12, then 80 steps of coasting; the ball must end inside the goal disc at (6.6, 2.5), radius 0.5 m. A planner with budget K draws K candidate nudges from the disc |a| ≤ 3 m/s, asks a model for the ending of each, and executes the one with the smallest imagined miss in the real Courtyard (lesson 9). A model's decision score is the share of 100 fixed launches on which that push ends in the goal: the exam that matters, and the only one that costs real launches. With the simulator as the model it is the ceiling every model shares: 6 % at K = 1, 43 % at 16, 85 % at 64, 100 % at 128.

A model here is lesson 8's kind: state at the nudge and nudge in, ending out; a ridge regression on M random tanh features, fitted in closed form to a log of n real trials, with E = 4 bootstrap members (lesson 5) whose mean is the answer. It trains in milliseconds, so twenty zoos of twelve are cheap.

Rank the twelve by the exam everyone runs first, the error on held-out trials from the model's own log. narrow is first with 0.28 m; big is fifth with 0.70 m, 2.5 times worse. Rank them by the decision at K = 64: big scores 45 %, narrow 24 %. The model with the smaller error decides about half as well. Over twenty zoos, each retrained from its own logs, the lowest held-out error is never the best decision (0 of 20 agree); narrow has the lowest error in 19 of them, big the best decision in 13. Lesson 15's decoy was one such pair: its smooth networks beat a model that had never met a wall on one-step error, and lost to it on decisions.

2 · A ladder of exams, each scored against the decision

A zoo model answers in one jump, so its one-step and rollout errors coincide and it has no code to probe; rungs 2 and 3 appear in the atlas of §5. Five exams apply to the zoo, in the order of the exam slider: held-out error, coverage, intervention error, imagined success, judged success.

To score an exam against the decision, count the pairs of models that the two rankings order alike (c) and oppositely (d). Kendall's τ is (c − d) / (c + d), corrected for ties: 1 for the same ranking, 0 for none, −1 for the reverse; at τ = 0.4 a random untied pair is ordered as the decision orders it with probability (1 + 0.4)/2 = 70 %. Over twenty zoos (K = 64, the series' goal) the five exams have median τ 0.38, 0.29, 0.25, −0.08 and 0.05. No cheap exam is far above 0.4, and the one that sounds most like the decision, the success the model imagines for its own plan, is the worst. Real launches do better. Against the success on a thousand other launches (twelve zoos), ten score τ = 0.58, twenty 0.71 and two hundred 0.89, where the held-out error, the best cheap exam, scores 0.45 (0.38 over the twenty).

The field met the same thing. A human study "confirms that FVD correlates well with qualitative human judgment" (FVD; Unterthiner et al., 2018), yet FVD "increases only slightly with large temporal corruption" (Ge et al., CVPR 2024). Video generators' "physical understanding is severely limited, and unrelated to visual realism" (Physics-IQ; Motamed et al., 2025). In a closed loop "visual quality alone does not guarantee task success, controllability matters more" (World-in-World; Zhang et al., ICLR 2026), and WorldArena (Shang et al., 2026) reports "a significant perception-functionality gap". Not every cheap benchmark fails: DreamGen Bench (Jang et al., 2025) "shows a strong correlation between benchmark performance and downstream policy success". Whether a cheap exam tracks the decision depends on the exam and the use (§3).

3 · Why the rungs disagree: the log, the use and the planner's search

The exam samples from the log; the planner does not. A held-out trial is drawn from the model's own recipe, so its nudge is one the log could hold. The planner draws over the whole disc and keeps the nudge the model likes best, so it asks where the log did not go: lesson 7's seeing against doing. The intervention error measures that, on 300 nudges set by fiat over the whole disc.

For big, whose log covers the disc, the two errors agree. For narrow the second is 33 times the first: 88 % of its picks lie beyond the largest nudge in its log, and there it hits 16 times in 88, against 8 in 12 inside. confounded is the sharper case: third lowest held-out error, worst error under nudges set by fiat. If the operator pushes against each gust (lesson 7), a = u − κw (u a random nudge of spread σ_u = 0.15 m/s, w a gust of spread σ_w = 0.05 m/s², κ = 5.76 s the nudge that cancels a unit gust), the ending is f(s) + Bu (f the ending without a nudge): the gust cancels out of it, and a regression of ending on nudge recovers only the fraction σ_u² / (σ_u² + κ²σ_w²) = 0.213 of the true effect B on an open floor (40,000 simulated trials: 0.213) and −0.04 in the walled Courtyard. The log says that nudges do nothing. The model fitted to it has a held-out error 3.1 times below the reference's, 30 times larger under nudges set by fiat, and a real success of 13 %. Over twenty zoos both narrow and confounded beat the reference on held-out error in 20 of 20, and in 20 of 20 the intervention error exposes them (more than five times their held-out error).

The exam is for a use. Move the goal from (6.6, 2.5) to (4.0, 1.2). Of the nudges that hit the series' goal, 28 % are at most 1.2 m/s, the reach of a cautious log; of those that hit the moved goal, 1.8 %. So cautious falls from 36 % to 10 % (median of twenty zoos, K = 64) while big goes from 43 % to 41 %, and the exams reshuffle. The median τ of the held-out error goes from 0.38, the best of the five, to −0.15; that of the intervention error goes from 0.25 to 0.76. An exam is valid for a use to the extent that it asks what the use will ask. The intervention error asks what a planner asks, of the whole disc, and stays positive for both goals where the held-out error does not.

The planner finds the holes. The success a model imagines for its own plan is the exam that sounds right, and at the series' goal the weakest. Among 128 candidates the planner keeps the one with the smallest imagined miss, the one whose error flatters most: lesson 8's optimiser's curse. At K = 128 a zoo model imagines a hit on 91 % of launches on average and really hits on 26 %; raising K pushes the imagined number towards 100 % for every model, so it ranks none. An exam can hold out the planner: plan with two members of the ensemble, let the other two grade the pick. The judged success averages 66 %, closer and still flattering, because the members share their features and log, and so their mistakes.

best = Infinity; bi = 0;
for (k = 0; k < K; k++) {
  q = (i * K + k) * 2; m = Math.hypot(P[q] - g.x, P[q + 1] - g.y);
  if (m < best) { best = m; bi = k; }
  if (Math.hypot(S.TE[(i * K + bi) * 2] - g.x, S.TE[(i * K + bi) * 2 + 1] - g.y) < g.r) real[k] += 1 / N;
  if (best < g.r) imag[k] += 1 / N;                                           // what the model said it would do
}
...
a = Math.sign(x[i] - x[j]); b = Math.sign(y[i] - y[j]);
if (a === 0 && b === 0) continue; if (a === 0) tx++; else if (b === 0) ty++; else if (a === b) c++; else d++;
return (c - d) / Math.sqrt((c + d + tx) * (c + d + ty));

What to try. It opens on the held-out error, K = 64, the series' goal, narrow (A) against big (B). (1) τ = 0.46 for this zoo; the star marks narrow, the best model on this exam, and the readout puts the pair side by side: 0.28 m and 24 % against 0.70 m and 45 %. (2) Step the exam slider through the five exams; the second readout lists their τ in order, and imagined success is the lowest, −0.11. (3) Switch to the moved goal: the five become −0.23, 0.20, 0.72, 0.69, 0.56, and narrow falls to 1 %. (4) Back on the series' goal, raise K to 128: the imagined curves of A and B reach 69 % and 97 %, the real ones 26 % and 50 %, the exact model 100 %. (5) On launch #40, A imagines landing 0.04 m from the goal's centre and lands 2.20 m away; B imagines 0.16 m and lands 0.41 m, a hit. (6) twin against mid, one recipe with other luck: 26 % against 19 %; telling them apart would take 558 launches per policy.

4 · What the decision costs

A success rate measured on n launches has standard error √(pq/n), q = 1 − p: 5.0 points at p = 0.45 and n = 100. To tell two policies apart, the gap must stand out from the error of the difference, whose variance is 2p̄q̄/n if the policies are equal and (p₁q₁ + p₂q₂)/n if not. A 5 % test rejects beyond z_α/2 = 1.96 standard errors of the first; it has power 80 % when the true gap clears that threshold by z_β = 0.84 standard errors of the second. Solving for n per policy:

n = ( z_α/2·√(2p̄q̄) + z_β·√(p₁q₁ + p₂q₂) )² / (p₁ − p₂)²

For 55 % against 40 % that is 172.8, so 173 launches per policy. Simulated at that size, the gap is found in 79 % of experiments (target 80) and two equal policies are called different in 4.7 % (target 5). The widget's pair, 24 % against 45 %, needs 80.

A model is a sample. Retrain the reference 60 times, each with its own log, features and bootstrap, and measure each on 900 other launches: real success runs from 13 % to 49 %, mean 27 %, standard deviation 7.9 points, 1.8 times the 4.4-point binomial error of a 100-launch benchmark. Two single training runs differ by that spread: the twin differs from the mid by 9 points (median over twenty zoos), by up to 33. The cheapest protection is the control the zoo carries: train the reference twice and believe only gaps larger than the A/A gap, the gap between a recipe and itself.

The winner of a benchmark is flattered. Pick the best of the twelve by its score on n launches and measure it on other launches (every disjoint block of a thousand fresh launches, twelve zoos, K = 64):

The second row is lesson 8's law with models for candidates: for true successes with spread σ (13 points here) measured with noise δ = √(p̄q̄/n), the winner of 12 is flattered by δ²/√(σ² + δ²) · E[max₁₂], E[max₁₂] = 1.63 being the expected maximum of twelve standard normals.

Zero failures proves less than it sounds. If the true failure rate were p, n launches would all be clean with probability (1 − p)ⁿ. The largest p that leaves this at least 5 % solves (1 − p)ⁿ = 0.05, that is p = 1 − 0.05^1/n. Since ln(1 − p) ≈ −p for small p, p ≈ ln 20 / n = 3.00 / n: the rule of three, that zero failures in n launches bound the failure rate at about 3/n with 95 % confidence. After 100 clean launches the exact bound is 2.95 %, after 300 it is 0.99 %. By simulation, with the rate at the 100-launch bound the launches are clean in 4.9 % of 40,000 runs; at 2 % in 13.5 % (exactly 13.3 %). Showing a rate below 1 % takes 299 clean launches, below 0.1 % 2,995.

A model can judge other policies, not its own. When launches are scarce a world model ranks policies. 1X did so for its Redwood checkpoints (16 June 2025): "Given a true real-world success rate gap of 15% between two policies, a World Model with 70% accuracy can accurately predict the better policy with 90% success." One reading, ours and not theirs: the model's verdict on an episode is right with probability a = 0.7 whoever the policy is. The gap then shrinks by 2a − 1 = 0.4, 55 % against 40 % is seen as 52 % against 46 %, and a 90 % chance of ranking the pair right takes about 227 judged episodes per policy (simulation at 227: 89 %). In the zoo each model grades the plans of the other eleven with median τ = 0.49 and a mean bias of −2.7 points, but grades its own plan 58 points too high. Gemini Robotics in a Veo World Simulator (11 Dec 2025) reports accurate prediction of relative policy performance in nominal and out-of-distribution conditions, validated by over 1,600 real evaluations of 8 checkpoints on 5 bimanual tasks: 40 per checkpoint and task if spread evenly, where a 15-point gap between two cells is detected 27 % of the time. The validation is about ranking across many cells, not a certificate for one.

5 · The failure atlas: ten failures, two exams each

Ten failures of lessons 3 to 15 each have a row: a cheap exam that the broken model passes and another that catches it. Rows 5 and 6 are read off the zoo above. The others are small separate runs of that lesson's experiment, not part of the widget: lesson 3's instruments (rows 1–2), the fork of lesson 5 (3), a one-step model fitted to 150 launches (4, 10), the exact model with a short-sighted planner (7), windows of readings behind a 1.6 m curtain (8), two identical balls (9).

The cheap exam is never wrong; it answers a question the failure does not touch. The atlas is not the whole series: a force a camera cannot see and a goal given in words (lesson 15) have no analogue in a ball-in-a-yard zoo, and the contrast between a world model and a policy is not tested here; none has a row.

6 · Design by product, and what to measure first

The one real push succeeds only if some candidate hits and the model's favourite is one of the hits. The first factor, reach, belongs to the task and the budget: the exact planner has 85 % at K = 64. The second, selection, belongs to the model: real success divided by reach. big picks a hit 53 % of the time when one is on offer, narrow 28 %, so 45 % = 85 % × 53 % and 24 % = 85 % × 28 %. Real systems are chosen on more factors: the horizon over which a model can be trusted (lesson 6), how well it obeys the action (controllability, lessons 11 and 14), the fidelity where the decision is made, the cost. When factors multiply, a small one caps the product however large the others, and an exam of one factor cannot rank systems that differ in another.

What does better selection cost? Capacity and data, at K = 128 (mean of three zoos; the exact model scores 100 %). A trial is 81 steps, 8.1 s of world time, plus a reset.

Neither alone is enough: sixteen times the data lifts 32 units from 17 % to 23 %, eight times the units lift 250 trials to 41 %, both together to 81 %.

What to measure first. (1) Compare the inputs the use will present with the range of the log. (2) Run the exam that samples those inputs, the intervention error if the planner will leave the log. (3) Grade plans with models that did not choose them. (4) Spend real launches on the finalists, 173 per policy for a 15-point gap, once the A/A control has shown the noise floor. Never rank on imagined success; use the held-out error only to choose between retrains of one recipe.

Common mistakes / failure modes

Checkpoint exercise

Where this points next

The ladder says how good a model is, and the price list says what knowing costs. The best model in the zoo succeeds on 45 % of launches at K = 64 where the exact model manages 85 %. No exam closes that gap: a better model is found by training, and the grid shows that how large, on how much, and from where move the decision from 17 % to 81 %, with 256 units and 9 hours of world time. Everything in this series assumed that a trained world model exists. How is one trained: which loss, which schedule, which data, and what does it cost?

Interview prompts

• Why can the model with the lowest held-out error make the worst decisions? (§1, §3 — the held-out set comes from the log, the planner asks over the whole action range.)
• What does Kendall's τ say that a correlation does not? (§2 — agreement of orderings, which is what choosing needs: (1 + τ)/2 of random untied pairs are ordered alike.)
• Why is a model's imagined success a bad exam, and what repairs it? (§3 — the planner keeps the candidate whose error flatters, the optimiser's curse; grade with models that did not choose.)
• How many launches tell two policies 15 points apart? (§4 — the power formula: 173 per policy for 55 % against 40 %.)
• A policy passes 100 launches with no failure. What can you claim? (§4 — (1 − p)¹⁰⁰ = 0.05 gives p ≈ 3 % at 95 % confidence: the rule of three.)
• Why is a benchmark's winner flattered, and why train the reference twice? (§4 — it is selected on the noise, δ²/√(σ² + δ²) · E[max]; the A/A gap is the noise floor.)

Companion reads: Lesson 30 · Training-side evaluation (the same exams during training), Robot Model Training · 15 Evaluation you can afford (launch budgets for robots) and Synthetic Vision · 11 What braking changes (the closed-loop harness).