World-model trajectories and closed-loop evaluation

A world model is useful only when its imagined consequences improve decisions. Build the evidence in causal order: preserve history, intervene on actions, represent uncertainty, test rollouts, then let a planner try to break the model.

Learning objectives and running case

By the end, you should be able to specify a trajectory dataset, explain when an action comparison is causal, choose a prediction representation, define every evaluation score, diagnose rollout failure, and design a hybrid simulator–world-model system. We will carry one case through the lesson:

The linear chain is:

Skipping a link creates a familiar illusion: sharp video without correct control, low one-step error with unstable rollouts, calibrated average risk with catastrophic slices, or excellent simulator results that do not transfer.

0 · The fundamental problem: prediction matters through decisions

A future prediction is not valuable merely because it resembles what happened. It is valuable when it preserves the distinctions that change a decision. Let a model assign a predicted distribution q(y|h,do(a)) to decision-relevant outcome y after history h and candidate action a. A rational consumer chooses:

Read this inside out. C(y,a) says what the outcome and action cost. The expectation combines possible outcomes using the model's probabilities. The minimum selects among feasible actions A(h). The model is useful only insofar as the selected action has low cost under the real process p. Its decision regret is:

This exposes two facts that image-quality language hides. First, a large prediction error may be harmless if it cannot change an action: the precise texture of the van usually does not alter braking. Second, a visually tiny error may be catastrophic if it crosses a decision boundary. Moving the predicted pedestrian occupancy 0.6 m sideways might barely affect a video similarity score yet change “our paths intersect” into “safe to coast.” A world model can therefore generate a believable movie while giving the planner the wrong answer.

The central identification problem follows. Passive logs tell us what followed actions selected by the logging policy: p_πb(y|h,a). Planning asks what would follow each alternative action we might deliberately choose: p(y|h,do(a)). These are different questions whenever the policy's action carries information about hidden danger, or whenever an action was never tried in that context. Visual prediction under the recorded future does not close this gap.

Everything else in this lesson is forced by that problem rather than added as a checklist:

1 · The unit of evidence: frame → transition → episode

Why a frame is insufficient

A frame dataset supports questions such as “which pixels belong to the pedestrian?” A world model must answer “if the shuttle brakes now, what happens over the next four seconds?” One image does not identify pedestrian velocity, shuttle acceleration, actuator delay, occluded intent, or whether a radar return belongs to the pedestrian. Even two visually identical frames can imply opposite safe actions because their histories differ.

Let o_t be the measurements available at decision time, a_t the applied action, s_t the complete simulator state, and c_t a cost. The data-generating process is:

Here h_t=(o_0:t,a_0:t−1) is the observable history. A transition preserves an input state/history, action, and next outcome. An episode preserves an ordered sequence of transitions, timing, termination, and lineage. Frames may still be sampled for image training, but the trajectory record must remain authoritative.

What the episode must preserve

episode_id, scenario_id, generator_version, split
branch_family_id, branch_id, parent_checkpoint_id
state_time, decision_time, action_command_time, action_apply_time
camera_exposure_start/end, lidar_beam_times, radar_scan_time
observations and calibration references
authoritative ego / pedestrian / vehicle / map state
action_command, action_applied, behavior_policy, action_probability
contacts, near-miss, collision, cost, termination, valid_mask
exogenous_seed_group, per-process seeds, provenance, QA status

Do not collapse timestamps into a single frame number. In the case study, the lower rows of a rolling-shutter image may be captured tens of milliseconds after the upper rows, and LiDAR points come from a scan interval. The command time and actual braking time differ by 0.25 s. If training aligns all of them to the same instant, the model can “learn” false acceleration or predict effects before actions occur.

2 · Hidden state, history, and belief

The physical simulator can expose s_t, but a deployed model sees only measurements. The parked van hides the pedestrian's legs; one image cannot reveal whether the pedestrian will stop. The proper inference target is therefore often a belief:

A belief is a distribution over plausible current states, not a single best guess. A learned summary z_t=E(h_t) is useful when it preserves the parts of that belief needed for future outcomes and decisions. “Latent state” does not automatically mean physical state: it may mix geometry, intent, appearance, and model artifacts. Test its sufficiency rather than naming it.

A state is Markov sufficient for a prediction if the future is conditionally independent of earlier history once that state and future actions are known. A practical test is to train a predictor from z_t, then ask whether adding older history consistently improves held-out likelihood or decision quality. Improvement suggests missing memory; no improvement does not prove true sufficiency because the test model may be weak.

Derive what the representation is allowed to forget

Compression is safe only relative to the promised decisions. Imagine two histories h and h′ that look identical now. In the first, radar has shown the pedestrian accelerating for 0.4 s; in the second, radar has shown deceleration. If there exists any feasible action sequence for which their future collision distributions differ, an encoder must not map both to the same state. Once merged, no downstream planner can reconstruct which history occurred.

This is decision-relative predictive sufficiency. It explains both memory and abstraction. History is required because current pixels alias different causal states. But a representation need not retain van paint texture if changing that texture never changes the promised outcomes under any action. “More information” is not the objective; retaining every decision-relevant distinction is.

3 · Observed actions are not automatically interventions

The confounding problem

Suppose expert drivers brake mainly when danger is high. In passive logs, collision frequency after braking may exceed collision frequency after coasting. It would be absurd to conclude braking causes collision: danger caused both the action and outcome. Observational data estimates p(s′|h,a) under a behavior policy. The causal query is what would happen if we forced a feasible action while holding the pre-action situation fixed:

If s truly contains every common cause of action and outcome, the observational conditional can equal the interventional distribution. In practice the learned history representation may omit intent, friction, or driver knowledge, so that equality must not be assumed. A simulator provides a cleaner experiment: save complete state before the decision, restore it, replace only the action, and roll forward.

Positivity: an effect cannot be learned where no alternative was tried

Even after controlling every common cause, observational identification requires positivity, also called action support: for every context where the model will compare an action, the behavior policy must assign that action nonzero probability. In discrete form, π_b(a|h)>0. For continuous controls, training needs density in a meaningful neighborhood, not the exact probability of one floating-point command.

Consider a deterministic expert that always brakes when its private hazard detector fires and always coasts otherwise. The log contains no “coast while detector fires” transition. Two mechanisms fit every logged trajectory equally well:

• Mechanism A: braking prevents collision; coasting in hazardous states would collide.
• Mechanism B: the hazard would disappear by itself; coasting in those states would also be safe.

Both predict the observed expert future perfectly because the missing action is never observed. No amount of passive data from the same deterministic policy distinguishes them. This is not a small-sample problem; it is missing evidence. A structural physics assumption might prefer A, and a simulator can safely branch the missing coast action, but the conclusion then depends on that structure or simulator validity.

Positivity is also representation-dependent. A dataset may contain many brake and coast actions globally while containing only brake near occluded pedestrians on wet roads. Report support jointly over state/history, action, and horizon. Do not “solve” lack of support by generating physically impossible steering or braking: an intervention answers the deployment question only if it belongs to the feasible action set. Finally, a perfect branch conditioned on privileged simulator state does not guarantee a deployed encoder can distinguish that state. Train and test the observation-to-belief link too.

Matched action effect

For outcome g, define the conditional action effect between brake and coast:

When g=1 means collision, negative Δ means braking reduces risk. This estimate is valid only for the specified checkpoint distribution, feasible action definitions, horizon, and disturbance distribution. It is not a universal effect of braking.

Numerical example

From each of 200 checkpoint restorations, generate a brake and coast branch with the same pedestrian-intent, friction, and latency draw. Collisions occur in 8 brake branches and 138 coast branches:

The estimated risk reduction is 65 percentage points. Because outcomes are paired by disturbance, estimate uncertainty from the 200 within-pair differences, not by pretending the 400 outcomes are independent. Also report absolute counts: a rate without sample size hides uncertainty, and 0/20 does not establish zero risk.

A kinematic check explains the direction. During 0.25 s latency the shuttle travels 2 m, leaving 16 m. Braking from 8 m/s at 4 m/s² requires v²/(2|a|)=8 m, so nominal stopping distance is 10 m including latency. Eight metres of margin remain before the original crossing line. Wet friction, a late response, or pedestrian motion can consume that margin—hence 4%, not zero, collision risk.

4 · Checkpointed branches: what should share randomness?

A branch family contains alternative futures restored from one checkpoint. It isolates action effects and is more sample-efficient than hoping two unrelated episodes happen to match. The implementation must make the comparison scientifically meaningful.

Common random numbers

For a matched pair, reuse pre-specified exogenous draws that should not change merely because the action changed: pedestrian intent sampled before the intervention, road friction, wind, initial sensor offsets, and base reaction-time draw. This is called common random numbers. It lowers comparison variance because both actions face the same difficult or easy world.

Do not force action-dependent consequences to match. Once braking changes the scene, contact impulses, occlusion, sensor returns, and an actor responding to the shuttle may legitimately diverge. Separate random streams by named process rather than sharing one global generator: otherwise one branch making an extra physics call shifts every later random draw, producing accidental differences.

The first-principles view uses potential outcomes. Let Y(a,ξ) be the outcome under action a and exogenous circumstances ξ. A matched contrast evaluates Y(brake,ξ)−Y(coast,ξ) for the same ξ. Independent draws instead compare Y(brake,ξ₁)−Y(coast,ξ₂), mixing the action effect with weather, intent, and latency differences. Sharing valid pre-action randomness removes that avoidable variance; it does not require downstream trajectories to remain visually identical.

seed_family = hash(checkpoint_id, replicate_id)
seed_friction   = hash(seed_family, "friction")
seed_ped_intent = hash(seed_family, "pedestrian_intent")
seed_latency    = hash(seed_family, "actuator_latency")
seed_sensor     = hash(seed_family, "sensor_noise", branch_id)
# Sensor noise may be shared for a pure action comparison, or independent
# when estimating the observation distribution. Record the choice.

Independent disturbances

Matched branches estimate a low-variance contrast. Independent repetitions estimate the full distribution the model must represent. You need both:

• Matched phase: same checkpoint and common exogenous draw, different action. Measures sensitivity and action ranking.
• Replication phase: repeat each action with independent draws. Measures collision probability, tail stopping distance, and multimodality.
• Shift phase: deliberately vary friction, intent, lighting, sensor faults, and latency. Measures robustness and extrapolation by named slice.

Log whether a variable is pre-treatment, action-responsive, or merely measurement noise. Accidentally controlling an action mediator—such as forcing the pedestrian to follow an identical path despite visibly reacting to braking—can remove a real causal pathway.

5 · Design temporal evidence for identifiability

A parameter is identifiable when the available observation–action evidence distinguishes its possible values. More frames do not automatically help. Ten seconds of a stationary shuttle may reveal camera noise but not tire–road friction; a brief controlled brake pulse may reveal friction immediately.

1. Name the hidden variable. For example pedestrian velocity, intent, friction, or delay.
2. Construct competing worlds. Choose two plausible values that look the same initially.
3. Ask what sequence separates them. More history may reveal velocity; an action probe may reveal friction; intent may remain irreducibly uncertain.
4. Render through the actual sensor. If dusk noise erases the distinguishing evidence, state labels alone do not make the inference learnable at deployment.
5. Keep unresolved alternatives. If evidence cannot distinguish “wait” from “cross,” supervise a distribution rather than a fabricated deterministic label.

Vary context length, observation gaps, sensor dropout, action delay, and prediction horizon. Include belief-correction episodes: hide the pedestrian for one second, let the model roll open loop, then reveal a clean radar or camera observation. Measure how quickly its belief recovers. This tests filtering, not just extrapolation.

6 · Choose the prediction representation from the decision backward

There is no universally correct world-model output. Start with the planner's query and preserve the cheapest representation sufficient to answer it.

Action-free video can learn visual invariances, persistence, and passive dynamics, but cannot identify the consequence of a specific brake command. V-JEPA 2 (Meta, June 2025) exemplifies a useful division: large-scale action-free video representation learning followed by an action-conditioned robot world-model stage. This is an architectural example, not proof that one representation dominates every task.

For the shuttle, a sensible hybrid predicts probabilistic pedestrian occupancy, ego state, and collision events, then optionally decodes sensor futures for diagnostic visualization. A photorealistic video decoder is not required for emergency braking if occupancy and uncertainty preserve all decision-relevant information.

7 · Uncertainty and rollout stability

Three uncertainties

• Aleatoric: irreducible future variability, such as whether the pedestrian hesitates. Represent it as a distribution or multiple weighted futures.
• Epistemic: limited knowledge because training lacks wet dusk crossings. Estimate with ensembles, posterior approximations, or disagreement, and verify that it rises on meaningful shifts.
• Model misspecification: the representation cannot express the truth—for example a single Gaussian over stop/cross modes. More data alone may not fix it.

For a binary collision event with predicted probability q and outcome y∈{0,1}, the Brier score is the mean (q−y)²; lower is better. Negative log-likelihood is −[y log q+(1−y)log(1−q)]; it strongly penalizes confident errors. Both are proper scoring rules when computed on the declared outcome and distribution, but NLL requires probability clipping for numerical stability and is sensitive to density parameterization for continuous states.

Calibration means events predicted at probability q occur about fraction q in the relevant population. Expected calibration error (ECE) bins predictions and averages |accuracy−confidence|. Report bin definitions and counts: ECE can look small with coarse bins or cancel slice failures. Also report a reliability table/curve and safety-critical slices.

Example: among 100 wet-dusk coast branches assigned 0.20 collision probability, 41 collide. The bin calibration gap is |0.41−0.20|=0.21: severe risk underestimation. Recalibration may align scores on known data, but it cannot add a missing causal mode or guarantee behavior under shift.

Why rollout error compounds

One-step evaluation often feeds true state at every step. Deployment feeds the model its own prediction. If transition error is ε and the learned dynamics amplify state differences by factor L, a rough bound after H steps is:

This is a diagnostic bound, not an exact law. When L>1, small errors can grow rapidly; contacts and visibility create discontinuities that a smooth bound may not describe.

Suppose shuttle position RMSE is 0.08 m at 0.5 s, 0.22 m at 1 s, 0.85 m at 2 s, and 3.1 m at 4 s. RMSE is the square root of mean squared coordinate error, so it emphasizes large misses and depends on coordinate frame. The curve reveals horizon collapse that a one-step score hides. Report mean displacement error over the trajectory (ADE), final displacement error at the horizon (FDE), collision-event metrics, and constraint violations; a low average position error can coexist with one fatal boundary crossing.

eOpen(H)   = ε                                 // teacher-forced: true state fed back
eClosed(H) = |L−1|<tiny ? ε·H                   // free-running: geometric accumulation
                        : ε·(L^H − 1)/(L − 1)
// L<1 ⇒ bounded limit ε/(1−L);  L>1 ⇒ explodes;  threshold 0.5 m flips brake/coast

Improve stability with multi-step training, rollout losses, physically structured updates, state projection, uncertainty growth, observation correction, and short receding horizons. Scheduled sampling can expose the model to its own states but changes the training distribution and does not guarantee consistent probabilistic learning. Always retest closed loop.

8 · Evaluation is a ladder, not a leaderboard number

What a model can cheat on

A score certifies only the information its test makes necessary. One-step training usually supplies the true current state, so a model can predict local motion without surviving its own mistakes. Video losses usually reward appearance under the recorded action, so a model can ignore the action when the behavior policy makes it predictable from context. Set-level perceptual scores can reward realistic color and texture without preserving the paired checkpoint, identity, geometry, event probability, or alternative-action ordering.

No finite metric proves universal control competence. The ladder works by removing one family of shortcuts at a time. Generator checks establish that the evidence means what it claims; one-step tests isolate local fitting; open rollouts remove teacher forcing; interventions force action use; uncertainty tests punish dishonest confidence; real tests challenge simulator assumptions; and closed loop exposes optimizer-selected errors. Later rungs do not make earlier diagnostics redundant—they help localize why the final system failed.

Precision is true predicted events divided by all predicted events; recall is true predicted events divided by all actual events; F1 is their harmonic mean. State the event window and matching rule. Interval coverage is the fraction of truths within a predicted interval; it must be paired with interval width, because an infinitely wide interval achieves perfect coverage. A risk–coverage curve sorts examples by uncertainty and plots error as the model abstains on uncertain cases; it is meaningful only if abstention is operationally possible.

Open loop versus closed loop

In open-loop evaluation, supply a fixed action sequence and compare predicted with true future. It isolates prediction and drift. In closed-loop evaluation, a policy replans from model predictions; its actions change future observations. This measures the combined model–planner system and exposes feedback failures.

Use identical initial-state and disturbance sets to compare controllers. Report confidence intervals across scenario families, not thousands of correlated frames. Keep collision and near-miss slices visible rather than averaging them with easy cruising.

Cost and regret example

With collision cost 1,000 and hard-brake cost 1, the true expected costs from matched evaluation are:

A correct planner brakes. Suppose an exploitable model predicts collision probabilities 0.06 for brake and 0.02 for coast, so it chooses coast because predicted costs are 61 versus 20. Its realized regret is the chosen policy's true cost minus the best candidate's true cost: 690−41=649. Regret is always relative to a stated action set or oracle; it is not an absolute property of the model.

9 · Planner exploitation: the strongest adversarial test

A planner searches many action sequences and selects the one with highest predicted value. This optimization magnifies rare optimistic errors. A model with excellent average video quality may invent a narrow “safe corridor” through the pedestrian because no training trajectory used a tiny steering oscillation. The planner will find it.

A simple calculation shows why average accuracy is weak protection. Suppose only 0.5% of candidate rollouts contain a dangerously optimistic error. On one random rollout, the model appears 99.5% reliable. If a planner evaluates 500 sufficiently varied candidates, the chance that at least one candidate contains such an error is 1−(1−0.005)⁵⁰⁰≈0.918, about 92%. The planner preferentially selects that false high-value candidate, so model errors in deployed plans are not random samples from benchmark error. Correlation between candidates changes the exact number, but not the selection-bias mechanism.

• Constrain search to dynamically feasible actions and deployment support; still test controlled extrapolation separately.
• Use uncertainty-aware or pessimistic value estimates, but verify uncertainty itself—ensembles can agree outside their shared training support.
• Adversarially search for sequences where predicted safety and authoritative rollout disagree.
• Use receding-horizon replanning so real observations correct belief before distant imagined errors dominate.
• Keep independent hard constraints and a safety shield for hazards the learned model must not own.
• Log the optimizer budget. A model safe under 32 sampled actions may fail under a million-query search.

Closed-loop success is necessary but not sufficient: a conservative policy can succeed by always stopping. Pair safety with progress, comfort, latency, and intervention metrics. Compare against simple baselines such as constant-velocity prediction and emergency stopping.

10 · Branching generator and evaluation harness

Dataset-generation pseudocode

for scenario in sample_scenarios(split="train"):
    episode = simulate_until_informative_event(scenario)
    checkpoint = save_complete_state(episode)

    for replicate in range(R):
        exogenous = sample_named_disturbances(scenario, replicate)
        for action in feasible_actions(checkpoint):
            restore(checkpoint)
            set_common_pre_action_disturbances(exogenous)
            command(action)
            trajectory = []
            for t in timeline(horizon=4.0, physics_dt=0.005):
                step_physics_and_actors()
                measurements = acquire_sensors_at_their_own_times()
                trajectory.append(export_state_action_measurement_events())
            write_branch(
                trajectory,
                branch_family_id=checkpoint.id,
                intervention=action,
                disturbance_manifest=exogenous,
                schema_version=SCHEMA,
            )

validate_replay_and_invariants()
split_by_scenario_asset_and_branch_lineage()

save_complete_state must include physics solver state, controller memory, actor beliefs, RNG stream states, and sensor timing—not just visible poses. Otherwise “restored” branches differ before the intervention. Verify restoration by replaying a no-change branch and comparing authoritative state within declared deterministic tolerances.

Evaluation-harness pseudocode

for family in held_out_branch_families:
    history = observations_before(family.checkpoint)
    belief = model.encode(history)

    for branch in family.branches:
        prediction = model.rollout(belief, branch.action_sequence, H)
        score_one_step(prediction, branch.truth)
        score_horizon_curve(prediction, branch.truth)
        score_events_constraints_and_uncertainty(prediction, branch.truth)

    score_matched_action_effects(family.predictions, family.truth)
    score_action_ranking(family.predictions, family.truth_costs)

for seed_set in closed_loop_seed_sets:
    run_same_seeds(model_planner, baselines, safety_shield)
report_overall_slices_confidence_intervals_latency_and_regret()
audit_planner_selected_actions_for_support_and_exploitation()

Freeze metric code, outcome horizons, coordinate frames, collision definitions, cost weights, and scenario slices before the final test. Version them like model code. A changed matching tolerance can improve F1 without changing predictions.

11 · Failure triage: diagnose before generating more data

12 · Operational state of the art: divide authority

“State of the art” is not one generator and is task-dependent. Operationally, the strongest pattern is a hybrid system with explicit ownership:

1. Structured simulation owns truth: geometry, identity, timestamps, actions, basic physics, annotations, checkpoints, and interventions.
2. Procedural scenario generation owns coverage: compositional layouts, behavior mixtures, long tails, and targeted failure families.
3. Neural digital twins or controlled video generation own difficult residuals: captured-place appearance, weather, materials, or dynamics that the engine cannot economically model—subject to structural controls and drift verification.
4. Passive real video supplies scale: representation pretraining learns observation structure; smaller action-labeled real and synthetic sets teach controllability.
5. Real held-out and closed-loop tests own acceptance: the generator never grades itself.

Why the hybrid is structural, not merely convenient

The modules have conflicting requirements. Exact labels and matched interventions demand persistent identity, deterministic replay, and inspectable state. Broad long-tail coverage demands procedural recombination and aggressive exploration. Realistic dusk appearance demands a flexible high-capacity observation model, which may move edges or invent content. Real data has the correct deployment physics and sensor process, but rare hazards and alternative actions are scarce or unsafe to collect. Asking one learned generator to satisfy all four requirements hides which assumption failed and lets the generator grade its own inventions.

Division of authority resolves the conflict. The engine supplies auditable causal structure; procedural programs distribute experiments across the support; learned residuals spend capacity where hand modeling is weakest; real measurements calibrate parameters the simulator can identify; and locked real/closed-loop tests reject the entire composition. The interfaces must still be tested: a generative refiner that shifts the pedestrian silhouette has crossed from appearance into label ownership, and a neural dynamics residual that overrides braking constraints has crossed into safety-critical physics. Hybrid is not automatically safer—it is stronger because ownership and violations can be made explicit.

Genie 3 (Google DeepMind, August 2025) demonstrates real-time interactive generated environments and reports limitations including action breadth, multi-agent interaction, geographic fidelity, and duration. It is evidence of rapid interactive-generation progress, not proof of exact physics or safety validity. WorldSimBench (Qin et al., 2024) usefully separates perceptual prediction from manipulative evaluation, where generated futures support robot action. Decision utility is the more demanding criterion. The action-conditioned generative branch is now productized — Cosmos 3 (NVIDIA, 2026) exposes a "Predict" model that generates future world states as video — while OmniDreams (2026) renders Gaussian-splatting reconstructions as a real-time closed-loop simulator for evaluating a driving policy, which is precisely the rung-7 test in §8.

13 · Design decision tree

1. What decision will consume the prediction? If none is specified, do not claim a control world model; define a representation-learning task instead.
2. What is the minimum sufficient future? Use state/occupancy for structured planning, sensor generation for sensor-level testing, or a verified hybrid when both matter.
3. Is the world partially observed? If yes, preserve history and output beliefs/multiple futures. If no, verify that the apparent full state truly contains hidden actors, delays, and parameters.
4. Are action effects identifiable in logs? If no, checkpoint and branch in simulation or run safe real experiments. Record behavior-policy support.
5. Is uncertainty aleatoric, epistemic, or representational? Use distributions for inherent branches, coverage/data for knowledge gaps, and change the output family for misspecification.
6. Will a planner optimize against the model? If yes, require adversarial action search, uncertainty checks, constraints, correction, and closed-loop replay.
7. What owns final truth? Synthetic QA validates evidence; frozen real slices and the target closed-loop system accept or reject it.

14 · Final build recipe

1. Write the task, sensors, feasible actions, horizon, costs, latency budget, and acceptance contract.
2. List hidden causes and the histories or interventions that identify each; retain distributions where identification is impossible.
3. Find informative checkpoints and generate expert, exploratory, perturb-and-recover, and risk-focused behavior with logged provenance.
4. Branch feasible actions with complete restoration, named random streams, matched disturbances, and independent replications.
5. Export asynchronous measurements, authoritative state, commanded/applied actions, events, costs, and branch lineage.
6. Split by scene, asset, scenario, and branch family; validate replay, causality, timing, and coverage.
7. Choose the cheapest decision-sufficient output; train one-step and rollout objectives with explicit uncertainty.
8. Evaluate the full ladder: generator, local transition, horizon, intervention, calibration, real transfer, and closed loop.
9. Let planners search for feasible exploits; add failures as parameterized training families without leaking the locked test.
10. Accept only measured reduction in held-out real cost at an acceptable compute, comfort, and safety budget.

15 · Exercises and self-test

1. Frame or episode? You need pedestrian segmentation only for single photos. Must every sample be an episode? No. Frames are sufficient for that narrow supervised task, but preserve episode lineage if you will evaluate temporal consistency or reuse the data for prediction.
2. Confounding. Logs show 12% collision after braking and 1% after coasting. Does braking increase risk? Not from these rates. Dangerous states cause braking. Compare actions at matched sufficient states or use an intervention.
3. Randomness. Should pedestrian hesitation be identical in brake and coast branches? If sampled before and unaffected by the observed vehicle, share the latent draw. If the pedestrian reacts to braking, the reaction is a causal mediator and may differ.
4. Calibration. Of 50 cases predicted at 0.10 collision risk, 8 collide. What is the bin gap? The observed rate is 8/50=0.16, so the absolute gap is 0.06. Fifty samples still imply substantial uncertainty; do not over-interpret one bin.
5. Coverage. A 95% trajectory interval covers 99.9% of truths. Is that automatically better? No. It may be too wide to support a decision. Report interval width or sharpness with coverage.
6. Rollout. Why can one-step RMSE stay low while collision prediction fails at four seconds? Teacher forcing hides self-generated state errors; small errors compound and collision is a discontinuous boundary event.
7. Planner test. Why replay the planner's chosen actions in the authoritative simulator? The planner selects optimistic model errors, so average held-out trajectories may not cover its induced action distribution.
8. SOTA claim. A video model produces realistic interactive dusk scenes. Is it a validated shuttle world simulator? Not yet. It must preserve action effects, geometry, uncertainty, sensor behavior, real transfer, and closed-loop decision utility.

Capstone system-design prompt

A strong answer does not begin with model architecture. It begins with the decision and failure cost, derives what future information is sufficient, asks what history makes that information inferable, then designs interventions and measurements. Only after the evidence contract is sound should it choose a network.

Primary sources and dated frontier

• Kubric — Greff et al., CVPR 2022; controlled scenes and rich vision annotations.
• BlenderProc — Denninger et al., introduced 2019; expanded journal version 2023; procedural Blender data pipelines.
• Infinigen — Raistrick et al., CVPR 2023; procedural natural worlds built from real 3D geometry.
• Structured Domain Randomization — Prakash et al., ICRA 2019; context-aware probabilistic scene graphs.
• Isaac Sim Replicator + Cosmos — NVIDIA documentation, accessed July 18, 2026; engine controls feeding generative refinement.
• Cosmos-Transfer1 — NVIDIA, March 2025; world generation controlled by multiple spatial modalities.
• UniSim — Yang et al., August 2023; neural sensor simulation from real driving logs.
• NeuRAD — Tonderski et al., November 2023 / CVPR 2024; dynamic neural rendering for camera and LiDAR.
• V-JEPA 2 — Meta, June 2025; video-pretrained representations with an action-conditioned robot world-model stage.
• Genie 3 — Google DeepMind, August 2025; real-time interactive generated environments and stated limitations.
• WorldSimBench — Qin et al., October 2024; perceptual and manipulative evaluation of video world simulators.
• Cosmos 3 — NVIDIA, 2026; world foundation models split into Predict (future states), Transfer (3D-sim → photoreal), and Reason (physics critic).
• OmniDreams — 2026; real-time (~105 FPS) generative reconstruction for closed-loop autonomous-vehicle policy evaluation.