Evaluation and failure modes

The final test is not whether a frame looks real. It is whether state stays coherent, uncertainty is honest, interventions have the right consequences, and decisions made through the model work safely in the world.

1 · Derive metrics from the decision, not the benchmark

The final exam for a world model is not “does the frame look real?” It is whether, when a planner acts through it, the robot succeeds safely in the real world — and a model can ace the first while failing the second. So start evaluation from the deployment decision, not a convenient benchmark. Define the action space, observation history, horizon, deadline, outcomes, and the asymmetric cost of mistakes; then work backward to the intermediate capabilities that decision needs.

Our running product is Countertop Assistant: place the instructed red ceramic mug on a tray within 20 seconds — correct object, upright, no spill, no human/scene collision, grip force under the material limit. The product loss might assign 1 to timeout, 3 to wrong object, 10 to spill or breakage, and 100 to human contact. Average pixel error contains none of those asymmetries, and even average task success can hide a rare but unacceptable safety tail. So write the operational objective explicitly:

Those thresholds immediately dictate what evaluation must measure: calibrated rare-event estimates or conservative bounds, correct response to action intervention, sufficient state for humans/contact/material, and tail latency. A photorealistic video metric can stay useful for debugging appearance, but it cannot be the release criterion.

2 · Climb an evaluation ladder instead of collapsing to one score

Isolate each arrow in the loop before testing the whole. Lower rungs are cheap and diagnostic; upper rungs are realistic but confounded and expensive — and passing a lower rung is necessary, not sufficient. Picture the first failed trial: the robot misses the mug. “The model is bad” isn’t an explanation yet; climb only after the rung below is known to work, because each rung turns a vague failure into one testable question:

0. Data: did the recorded cause actually precede the effect? Before judging a model, verify observations and actions are physically aligned — clock residual, reprojection error, missingness, action-effect lag. A failure here means dataset corruption or leakage; climbing further would only test how well a model fits a broken record.
1. Observation: did relevant evidence enter the representation? Once timing is credible, ask whether the model can encode and decode what the sensors saw — perceptual distance, depth and flow, contact/event timing, ASR grounding. Failure points to sensor representation, not yet memory or planning.
2. State: did momentary evidence become a persistent belief? Probe whether hidden variables stay recoverable — tracking, occupancy, pose, contact mode, conservation checks. If this fails, the belief is insufficient and later dynamics can’t predict a variable it discarded.
3. One-step dynamics: did one executed action cause the right local change? Measure state-transition error, mode likelihood, constraint violation. Failure after the state rung passes usually points to local dynamics or action alignment — the smallest action-conditioned unit a rollout is built from.
4. Free rollout: does the local rule survive the model consuming its own predictions? Plot error vs time, identity and constraint survival, event-time error. A gap between one-step and free-rollout quality localizes compounding drift rather than perception.
5. Uncertainty: does the model know when several futures remain possible? Check whether probabilities match frequencies and widen under ambiguity — NLL, Brier, ECE, coverage, sharpness, risk–coverage. Failure means dangerous overconfidence or useless diffusion.
6. Intervention: does a changed action change the correct outcome? Matched counterfactual error, treatment-effect error, controllability. A model that predicts recorded futures but fails here learned correlation, not causal dynamics.
7. Planning: does optimization through the model rank actions as reality would? Regret, top-k action recall, value error after selection, exploit rate. If intervention passes but planning fails, suspect planner–model mismatch or selection-amplified error.
8. Closed loop: does the assembled agent improve physical outcomes? Success, expected cost, collision, recovery, sample efficiency — sensors, action budget, and safety controller held comparable. This reveals integration failures individually-adequate parts can create.
9. System: does the same loop survive production load and faults? P50/P95/P99 latency, throughput, state corruption, fallback, availability. Failure here is an operating-envelope problem: a correct plan that arrives late or on stale state is physically incorrect.

The order is deliberately asymmetric: a high rung can fail because of anything below it, but a low rung doesn’t become valid just because the final task happened to succeed — one lucky grasp cannot excuse a reversed action timestamp, which will fail the moment speed or delay changes. So attach every release gate to the earliest rung that can explain it, and stop climbing as soon as a prerequisite breaks. A release dashboard should show this vector by horizon and scenario slice; a single scalar is useful for ordering experiments only after safety gates are enforced, because aggregate scores let excellent texture cancel incorrect braking and common success cancel a rare catastrophe.

3 · State probes test sufficiency, not visual beauty

A belief should make task-relevant futures predictable without the full past. Freeze the world model and train small probes for object identity/pose, occluded occupancy, joint state, contact mode, mass/friction bin, human motion, and goal predicates — a linear probe tests whether information is accessible, a powerful nonlinear probe whether it exists at all — and compare against probes on the latest frame and a simple history baseline. Probe under controlled ambiguity: hide the mug for two seconds and check identity and pose uncertainty; show visually identical dry and wet grasps, then add tactile evidence; swap instruction wording while keeping the goal; move the camera with the world static. (A model can score well on random states because labels correlate with backgrounds — counterfactual edits and held-out kitchens expose that shortcut.) Measure invariants and equivariances too: identity persists across viewpoint, geometry transforms predictably with the frame, rigid size doesn’t drift, free space and occupancy don’t overlap, contact forces roughly satisfy direction and magnitude. And for this product, don’t just probe “is the mug present?” — probe what the planner queries: mug pose covariance, tray support polygon, stable-contact probability, time-to-contact, human occupancy over the next second, and uncertainty when an observation is absent. If those aren’t recoverable, no planner hyperparameter can rescue the representation.

4 · Rollout evaluation is a curve over horizon and regime

One-step evaluation conditions on the true current state every step; deployment planning conditions on the model’s own predicted states, and small bias compounds. So evaluate free rollout: initialize from real history once, apply the recorded executed actions, and compare model distributions to the physical trajectory at every horizon. Report event-aware curves, not just average state MSE — position error can stay tiny while contact begins 150 ms early and flips the subsequent dynamics — so track object pose/velocity error, identity survival, contact-mode accuracy, collision false negatives, event-time error, reward/cost error, and constraint violations, bucketed by speed, contact, occlusion, material, lighting, robot revision, and whether the behavior policy is in support.

Numbers sharpen the point. If lateral pose has 4 mm one-step RMS error, an independent-noise story predicts roughly 4√20 ≈ 17.9 mm after 20 steps, while a steady 1 mm directional bias per step gives 20 mm plus noise. Measure 58 mm at 20 steps and the model is entering off-manifold states or switching contact modes wrongly — “one-step error × horizon” underestimates it. Plot posterior rollouts (which get new observations) separately from prior imagination (which does not). And multimodal futures need distributional metrics: a human may go left or right, so averaging invents someone walking through the middle — evaluate likelihood or proper scores of the realized future, prediction-set coverage, diversity without duplicates, and whether distinct samples are coherent modes, because best-of-N error alone rewards spraying samples and hides probability assignment.

5 · Test causal action effects with matched interventions

Passive prediction can succeed by learning what usually follows while ignoring the action — in demonstrations a gripper closes near a mug and the mug rises, and background plus operator intent predict most of it. A causal world model must answer what changes when the action changes with prior state comparable. For action a and outcome y, define the effect between two interventions:

The do means the action is set, not observed. Exact counterfactual pairs are impossible in one episode, so use resettable simulation, controlled fixtures, randomized safe interventions, or carefully matched initial states, and evaluate direction, magnitude, threshold, and interaction: more grip force reduces slip up to a point but may increase breakage.

Work an intervention example

Across 200 matched low-friction trials, a 6 N-per-finger grasp succeeds 78/100 and a 10 N grasp 92/100 — an observed effect of 0.92 − 0.78 = 0.14, a 14-point increase. The model predicts 0.70 and 0.96, an effect of 0.26, so its effect error is 0.12: right direction, exaggerated benefit, inviting the planner to over-squeeze. Now try 14 N: physical success drops to 0.88 because brittle mugs crack, yet the model predicts 0.99 — it learned a monotonic “more force is safer” shortcut from data that rarely exceeded 10 N. That failure is invisible in held-out passive trajectories and obvious under an intervention curve, so the release suite needs dose–response plots for speed, force, clearance, and information-gathering actions.

6 · Calibration asks whether confidence deserves trust

A forecast is calibrated if events predicted at probability p occur about fraction p of the time among comparable cases, and proper scoring rules reward both calibration and useful sharpness. For a binary event y∈{0,1} with predicted p, Brier score is (p−y)² and negative log-likelihood is −[y log p +(1−y)log(1−p)] — NLL punishes confident mistakes hard, right for safety but sensitive to label noise. Expected calibration error bins predictions and averages the confidence-vs-frequency gap; it’s readable but depends on binning and can hide subgroups, so always show reliability diagrams and calibrate by horizon, action selection, and safety slice — a model calibrated on random recorded actions can turn overconfident once a planner deliberately selects its highest predicted values.

Work a calibration example

Say 100 planned grasps each get predicted slip probability 0.10, but 24 actually slip. Group Brier is [24(0.10−1)² + 76(0.10−0)²]/100 = (19.44+0.76)/100 = 0.202, and the calibration gap is |0.10−0.24|=0.14. Had the same cases been predicted at 0.24, Brier would be [24(0.76²)+76(0.24²)]/100 ≈ 0.1824 — calibration improves the score, but the forecast is still uncertain: calibration is not accuracy. For continuous or multimodal futures, evaluate coverage and sharpness together — a nominal 90% pose region should contain truth ~90% of the time while staying small (a region covering the whole workspace is perfectly covering and useless) — plot coverage vs horizon and OOD distance, and separate irreducible aleatoric uncertainty from epistemic uncertainty, since the latter should trigger information-gathering or abstention.

7 · Risk–coverage turns uncertainty into behavior

An OOD or uncertainty number matters only if it changes what the robot does. Sort cases by uncertainty, abstain or fall back on the most-uncertain fraction, and plot error or safety cost on the rest — a good measure makes risk fall as coverage falls, and if dropping the “most uncertain” 20% barely changes error, the score isn’t ranking danger. Countertop Assistant can respond by slowing, shortening the horizon, choosing a larger-clearance path, making a test contact, requesting a new viewpoint, switching to a verified controller, asking the user, or stopping — and you must evaluate the whole selective policy, including the cost and safety of fallback: a fallback invoked on 60% of normal cases may be safe but defeats the product, while one invoked only after a collision is not protective. Choose thresholds on validation and freeze them for test, then report coverage, conditional success, expected task cost, a catastrophic-event upper bound, fallback frequency, and time-to-recovery — and single out false confidence (low uncertainty, high error), which is more urgent than merely uncertain cases.

8 · Closed-loop evaluation measures decision quality

Ultimately, run the model inside the intended planner and compare against a reactive policy, a no-model controller, an oracle-state bound, and ablations, with action budget, sensors, latency, and safety controller held comparable and enough seeds to estimate tails (report confidence intervals, not one success number). Metrics include task success, time, wrong-object rate, spill/breakage, minimum human clearance, peak force, recovery success, replans, information-gathering actions, energy, deadline misses, and fallback — and separate model failure (wrong consequence), planner failure (good model, poor search/objective), controller failure (plan not executed), perception/alignment failure, and requirement failure (the metric omitted what users care about). Run paired trials from identical initial conditions where possible: if world-model MPC lifts success from 82% to 90% but raises P99 latency so 4% of actions are stale, the integrated system may be worse. Closed-loop success alone is still not enough — a policy can exploit simulator defects or dodge hard cases — so combine task utility with the earlier mechanistic tests, real-world canaries, and scenario coverage; the ladder exists so a top-line number can’t erase a known causal flaw.

9 · Planning is an automated red team for exploitation

A planner optimizes whatever the model predicts, including its errors — and as optimization strength rises, selected actions drift away from typical data toward regions where a small optimistic bias is largest. This is the model-based analogue of adversarial examples and Goodhart’s law, so turn the planner into a test generator: search for trajectories with high predicted value and a warning sign — ensemble disagreement, low behavior-data density, constraint violation, extreme action, reward–observation inconsistency, uncertainty that paradoxically shrinks, or disagreement between structured and pixel heads — then validate the candidates in a trusted simulator, a fixture, or a bounded real experiment. In our product a planner might find that moving the wrist through the edge of an occlusion makes the model “forget” the mug while keeping task reward, or that a 16 N grasp is predicted perfectly safe because training support ended at 10 N. Record post-selection error — error on planner-chosen trajectories, not random test actions — and increase planner budget during evaluation, because a model that looks safe with 16 candidates may be exploited with 1,024. Mitigations (pessimistic ensemble values, uncertainty penalties, support constraints, conservative terminal values, short horizons with frequent correction, physics/safety constraints outside the model, adversarial collection, planner-aware training) all help, but none replaces external validation — if safety cost and reward share the same flawed representation, penalizing predicted safety cannot protect against an unrepresented hazard.

10 · OOD is not one axis

Distribution shift enters through appearance (new lighting), geometry (new mug shape), dynamics (wet surface), semantics (odd instruction), embodiment (worn gripper), sensor process (camera latency), action (planner extrapolation), policy (new collector), or combinations — so a “held-out test set” should be a matrix of these shifts at increasing severity, not one mystery bucket. The tempting mistake is to treat every unfamiliar input the same and attach one “OOD score,” but different shifts break different arrows, produce different evidence, and demand different behavior:

1. Covariate shift changes appearance, not physics. In a dark kitchen, reconstruction and state probes should degrade though mug dynamics are unchanged — the evidence says “the camera is less informative,” so lean on depth and touch, improve the view, and slow down, not invent a new friction law.
2. Dynamics shift changes what the same-looking action does. A wet mug or worn pad looks familiar while residual and contact disagreement rise after motion — a physical mismatch, so test-lift, lower acceleration, or regrasp; changing image augmentation would miss the broken causal relation.
3. Semantic shift changes which world state the user means. With “that cup,” the referent posterior should stay multimodal if several fit — unresolved grounding, so ask for clarification rather than moving confidently to the most common training referent.
4. Embodiment shift changes how commands become motion. New firmware or actuator delay shifts action-effect timing even in a familiar kitchen — detect the lag, recalibrate, and use only a verified controller until the mapping is re-established. This is why an action model can’t be versioned apart from its body.
5. Planner-induced shift moves action outside data support. An extreme grip has high predicted value because support is low — low density and ensemble disagreement are the warning, so constrain or abstain rather than trusting optimization to turn extrapolation into truth.
6. A novel hazard may not look uncertain at all. A transparent fragile vessel can be confidently mapped to a familiar wrong class — internal uncertainty may not warn you, so external rules, human review, and expanded data are the defenses when the hazard is absent from the learned state.

So the real requirement is stronger than “detect OOD”: identify which conditional relationship changed — sensor-to-state, state-and-action-to-future, language-to-goal, command-to-execution, or state-to-hazard — and pick the response that reduces that specific risk. The diagnostic is behavioral: under darkness seek better evidence, under wet contact test dynamics, under ambiguous language ask. One generic confidence threshold can’t prove those distinctions. And the detectors themselves need evaluation — AUROC/PR for known shift labels, but more importantly risk–coverage and closed-loop cost — remembering that near-OOD cases are harder than obviously alien ones because the model stays confident; test compound shifts, delayed detection, and whether state is reset or contaminated after a sensor glitch.

11 · Safety is an envelope, not a confidence threshold

Define the operational design domain: supported objects/materials, workspace, human proximity, speeds, sensors, robot revisions, latency hardware, and conditions. Outside it, do not imply capability; inside it, layer defenses — learned risk prediction, calibrated uncertainty, action and workspace constraints, independent collision monitoring, force/torque limits, emergency stop, safe fallback, and audit logs. For rare catastrophes, zero events in a small test does not prove safety: if zero failures occur in n independent trials, a rough 95% upper bound is 3/n (“rule of three”), so zero human contacts in 10,000 trials only supports a bound near 3×10⁻⁴ — far above a 10⁻⁵ requirement. Use accelerated simulation and adversarial scenarios for coverage, formal or verified constraints where possible, conservative canaries, and continuous monitoring — and be honest about dependence, since repeated near-identical trials carry less evidence than diverse independent ones. A safety case links hazard → detection/constraint → evidence → residual risk → fallback owner, and must test sensor loss, stale belief, packet reorder, model timeout, actuator saturation, human intrusion, planner exploitation, and recovery after stop — because a system that fails safely but can’t resume still needs an operational recovery design.

12 · System-design capstone: Countertop Assistant

In a design review, reason in a fixed linear order — never open with “use a Transformer” — and let each answer constrain the next:

1. Decision and deadline. 100 ms action chunks at 10 Hz; compare grasp/transport candidates over 1–2 s; keep a faster independent torque loop; define success and asymmetric safety costs.
2. Hidden state and observability. Track object identity/pose, occlusion, free space, human trajectory, mug mass/friction/contents, contact mode, body state, delay, and semantic goal — marking which are distributions, not point estimates.
3. Representation. Object slots plus local 3D geometry, a body/contact latent, semantic predicates, and a recurrent stochastic belief; keep RGB features only where identity/material needs them; don’t roll every pixel for planning.
4. Time and hierarchy. Align physical timestamps; visual scene at 12–20 Hz, body/contact faster, task plan at 1–2 Hz, reflex at 1 kHz; exchange action chunks, risk bounds, and interrupts.
5. Dynamics and outputs. Predict distributions over object/body transition, discrete contact mode, human occupancy, task progress, safety cost, and observation residual; use multimodal latents and an ensemble or posterior for epistemic uncertainty.
6. Data. Combine demonstrations, safe randomized interventions, varied simulation, failures/recoveries, and targeted real data; store executed actions, clocks, calibration, invalidity, robot revision, and behavior policy; split by kitchen, object instance, and time.
7. Training and compute. Pretrain modalities, learn short action-conditioned transitions, extend free rollout, add task/safety heads, train on planner-discovered failures; quote trajectory tokens, corpus tokens/FLOPs, rollout product, and P99 serving budget.
8. Planner/controller. A VLA proposes familiar chunks; risk-sensitive short-horizon MPC compares them; warm-start; replan after evidence; enforce workspace, clearance, force, and stop constraints independently.
9. Evaluation. Gate data alignment, state probes, horizon curves, contact timing, causal force/speed sweeps, calibration, risk–coverage, exploit search, closed-loop outcomes, OOD slices, and tail latency.
10. Deployment and recovery. Ship only inside a written envelope, shadow first, canary by scenario, log model/version/belief/fallback, monitor calibration drift, and roll back; when uncertain, view-change, test-contact, slow, ask, or stop.

A concrete tradeoff

A photorealistic video model preserves material cues and supports human-readable simulation, but 12-step diffusion across many candidates blows the 55 ms planning budget; a pure object/BEV model is fast but may discard transparency, wetness, and fine contact. So the chosen hybrid uses a rich encoder for belief correction, a structured low-dimensional latent rollout for 128 candidates, and a renderer only for debugging and a few finalist futures — trading universal video generation for decision throughput and causal inspectability. Likewise a five-second horizon captures a full pick-and-place but compounds uncertainty and compute, so short MPC over 1–2 s with a terminal value for longer goals and frequent real correction fits the product better; and an ensemble improves epistemic detection but multiplies rollout cost, so use two members for routine planning, a larger shadow ensemble for monitoring, and independent hard constraints for catastrophic risk. Every choice follows from decision value per unit latency and residual risk.

13 · Diagnose failures by tracing the loop backward

When a task fails, start from the physical outcome and work backward rather than staring at final video. Was the wrong action selected? Did predicted ranking match external outcomes? Was risk underpredicted or missing from the objective? Did the planner leave action support? Was belief wrong because evidence was missing, misaligned, or discarded? Was the correct plan late or mis-executed? Did fallback trigger and recover? Each symptom has a tempting fix that’s often wrong — use the contrast between what works and what fails to isolate the broken arrow:

1. Good one-step, bad long rollout: the local rule exists but can’t survive its own outputs. Compare teacher-forced and free-rollout curves and inspect mode-switch timing; if the gap opens only in free rollout, add free-rollout training and overshooting; if at contact switches, improve the state and its discrete modes. More image reconstruction wouldn’t touch the broken dependency.
2. Good random-action, bad planned-action prediction: selection moved the system into an optimistic error pocket. Plot post-selection error against support and disagreement; if error rises as support falls, constrain the planner, collect targeted interventions, or use pessimistic values. The model can be accurate on the test distribution yet unsafe on the distribution its own optimizer creates.
3. Good pixels, bad task state: the representation kept appearance and dropped decision variables. Probe geometry and contact and test causal action effects; failure calls for a task-state bottleneck and targeted objectives, because another perceptual-fidelity term reinforces the shortcut.
4. Calibrated overall, unsafe subgroup: averaging hid a conditional error. Draw reliability curves by material, horizon, and policy; if wet mugs or planner-selected grasps are overconfident, use group- and horizon-specific calibration and expand coverage — one global temperature can’t supply evidence absent from that subgroup.
5. Offline success, online failure: the prediction may be right while the streaming system changes its inputs or misses its deadline. Inspect latency, stale state, executed (not commanded) action, and concurrency; reproduce with streaming replay, define a deadline fallback, and fix the systems path before retraining.
6. Rising uncertainty without falling risk: the score isn’t connected to a protective decision. Run risk–coverage and replay the whole selective policy; if abstaining on high-score cases doesn’t reduce danger, learn a better score or threshold, and if abstention itself is dangerous, design a safer fallback. Merely displaying uncertainty changes no physical outcome.

The pattern is always physical symptom → controlled contrast → localized requirement → scoped response → regression test — which keeps a systems deadline-miss from being mislabeled a representation problem, or a planner exploit from being “fixed” by averaging in more ordinary clips. Keep each failure as a named regression scenario with raw trajectories, expected causal behavior, and versions, so incidents become durable knowledge and retraining can’t quietly reintroduce an old rare failure while improving the average.

14 · The whole series in one loop

Graphics supplies forward models from scene to sensor. Vision supplies evidence from sensor to state. 3D and 4D representations provide spatial persistence. Generative models represent multiple possible futures. Control and reinforcement learning turn predictions into decisions. Data and simulation systems preserve the trajectory and test it in closed loop; evaluation tests each causal arrow and the full closed loop. The world model is the contract that makes these disciplines describe one persistent, actionable world.

The linear dependency is the whole point. A planner cannot repair an unobservable state. More rollout compute cannot repair misaligned actions. Calibration cannot make an omitted hazard visible. A safety threshold cannot help if uncertainty does not rank failures. Closed-loop success cannot establish causality if the environment is too easy. Start from the physical decision, preserve information and time, predict interventions, expose uncertainty, plan within support, correct from reality, and measure what failure costs — which is exactly the line this whole series walked, lesson by lesson.

Interview prompts

• What is the minimum credible evaluation suite? Data/alignment checks, task-state probes, one-step and free-rollout horizon curves, uncertainty calibration, matched action interventions, planner post-selection error, closed-loop utility/safety, OOD risk–coverage, and tail latency.
• Why can a beautiful video model be a poor world model? Perceptual quality may preserve texture while shifting identity, contact timing, intent, or collision tails; decisions require causally correct task state and calibrated risk.
• How do you test whether the model uses action causally? Hold initial state comparable, deliberately vary executed action in simulation or safe trials, and compare predicted versus observed dose–response or treatment effects.
• What is the difference between calibration and accuracy? Calibration matches confidence to empirical frequency; it does not guarantee sharp or correct individual predictions. A broad forecast can be calibrated but unhelpful.
• How do you detect model exploitation? Increase planner optimization, find high-value low-support/disagreement trajectories, measure error after selection, and validate candidates in an external simulator or bounded real experiment.
• Why must OOD uncertainty trigger behavior? A score reduces no physical risk unless it causes information gathering, conservative control, abstention, or fallback; risk–coverage tests whether it ranks dangerous cases.
• How would you investigate strong offline metrics but weak deployment? Trace from outcome backward through executed action, planner ranking, belief and sensor alignment, then inspect stale state, deadline misses, concurrency, calibration shift, and fallback behavior.
• How should a world-model system-design answer begin? State the decision, deadline, horizon, unacceptable outcome, and hidden state; only then choose representation, dynamics, data, planner, compute, evaluation, and safety envelope.