Video world models

Compress observed trajectories, model futures that branch, and make actions actually control those futures — without mistaking a convincing movie for a faithful simulator.

1 · Start from the decision, not “the next pretty frame”

You’ve seen enough dashcam video to predict the next second of road fairly well. But “predict the next frame” and “be a driving simulator you can steer” are not the same thing — the first can ignore the steering wheel entirely. Anchor on the decision. A delivery robot nears a fork; the same camera prefix can be followed by three commands: steer left around a barrier, go straight and brake, or steer right into a side lane. A useful model must produce different consequences from the same past — preserving the barrier, other vehicles, layout, and speed; obeying the chosen control; allowing several pedestrian reactions; and staying uncertain where evidence is thin.

So the object to model is a conditional distribution over future trajectories, not a next pretty frame:

x may be pixels or compressed latents, a holds low-level controls or higher-level actions, g is goal/context, and m_t keeps facts that may no longer be visible. The output is distributional because other agents, hidden surfaces, and chance permit many futures. That one object quietly forces four separate questions: what survives compression (a codec sets the smallest event the dynamics can represent); how time is factorized (token order, block size, stride set compute and error growth); where control enters (alignment and conditioning decide whether alternatives really diverge); and what is remembered beyond the context window (long-lived identity can’t depend only on recent frames).

2 · The tokenizer sets the simulator’s resolution

Raw video is enormous — a 16-frame megapixel clip is tens of millions of values, and planning may need hundreds of rollouts — so most systems first compress video into a smaller grid or sequence. This is not neutral preprocessing: whatever the encoder throws away, the dynamics can never recover. A spatial codec maps each frame to a lower-resolution feature grid; a spatiotemporal codec maps a short tube of frames to tokens, compressing space and time together. Bigger patches and strides cut compute — but the robot may lose a thin bicycle spoke, a traffic-light change, or the exact instant a tire touches the curb, because a codec tuned for perceptual reconstruction happily drops details humans barely notice even when they decide the next state.

Two latent flavors, two temperaments:

• Discrete tokens: the encoder picks codebook IDs, giving categorical likelihood and the whole language-model toolbox. Compression is explicit — but a small codebook causes quantization error or dead codes, and one symbol change can jump the image discontinuously.
• Continuous latents: real-valued tensors where nearby codes mean nearby images, natural for diffusion, flow, or Gaussian transitions. Their density is harder to score directly, and an over-powerful decoder can paper over state inconsistency with smooth output.

The rate term rewards compression; reconstruction and perceptual terms reward visible fidelity. But a world-model codec needs an extra test the usual ones skip: can a probe recover object count, pose, speed, contact state, signals, and rare safety cues? Can the transition predict the next latent without inventing decoder detail? Check these before scaling the temporal model — a missing variable can never be repaired by a bigger transformer downstream.

3 · Temporal factorization sets speed, context, and drift

After encoding, the model has to turn a joint future distribution into computable pieces — and each architecture is really just a different answer to “how is the simulator allowed to reveal the future?” Derive them from that question rather than memorizing names:

1. Reveal one token after another: autoregression. The joint becomes ∏_k p(z_k | z_<k, a) — a direct likelihood, flexible context, natural for discrete tokens. The trap: an accurate next token doesn’t imply an efficient trajectory. One video frame is many tokens, so sampling takes many serial steps, and an early mistake enters the context and is then treated as fact. Measure wall-clock latency and open-loop error growth, not just next-token likelihood.
2. Reveal a region together: masked or block prediction. Hide a future block and iteratively fill it, updating locations in parallel with within-block context. But no true future exists at inference, so the mask schedule becomes part of the model — a poor confidence order can fill a late consequence before its early cause and break time. Diagnose by changing the schedule with weights fixed; big causal changes mean the inference procedure, not the learned distribution, needs work.
3. Start with a whole noisy trajectory and refine it: diffusion or flow. Global denoising can coordinate a lane change across every future frame and express several coherent modes — curing the blurry-average failure of deterministic regression. The cost is several network evaluations over the block, and a strong visual prior may still weaken exact action response. So you owe a quality–latency schedule plus action-contrast tests at every denoising budget.
4. Carry one compact state forward: recurrent latent dynamics. A single transition per step creates a belief bottleneck and cheap long rollouts — ideal when a planner compares thousands of futures. The compression that buys speed can discard fine detail or collapse diverse futures, so judge state and risk prediction before decoded texture; if the compact rollout is decision-correct, decode only selected candidates.
5. Separate slow intent from fast detail: a hierarchical hybrid. Slow layout, route, or keyframes evolve at low frequency while a lower level fills local motion — relieving one clock from covering both minutes of context and millisecond contact. The new failure is disagreement between levels (the plan says “turn right” while local frames drift left), so test cross-level consistency and action timing.

The Generative Continuous track derives diffusion and flow; the Computer Vision video track covers the observed-video side and shares this lesson's p(z_t+1:t+H | z_≤t, a_t:t+H−1) notation. The recurring tension is trajectory generation: denoising all future frames jointly coordinates a lane change better than predicting isolated frames, but each iteration adds latency, while autoregression reacts step by step yet a rollout of thousands of patch tokens can be far too slow to plan with.

4 · Action conditioning must be aligned, identifiable, and impossible to ignore

The robot’s steering command is not a caption — it’s an intervention applied over a known interval through a physical actuator, so represent it on the model’s clock. If controls arrive at 20 Hz and video tokens are four-frame tubes at 5 Hz, aggregate the controls or integrate them through a substep; a one-frame timestamp shift can teach “brake” to correlate with deceleration after it already happened. Actions can enter as interleaved tokens, block embeddings, cross-attended control sequences, or transition parameters — but the mechanism matters less than three tests:

1. Sensitivity: from the identical prefix and seed, changing the action must change the appropriate future variables.
2. Selectivity: steering should change ego path and induced reactions, not randomly alter weather, vehicle identity, or layout.
3. Magnitude and timing: the response should scale and arrive with realistic actuator delay, not merely match direction.

And beware the observational shortcut: drivers brake because a pedestrian appears, so “brake” correlates with pedestrians — a model can learn to render a pedestrian whenever it sees a brake token, reversing cause and effect (exactly lesson 07’s confounding). Diverse behavior, randomized or exploratory actions where safe, known policies, counterfactual pairs, and causal regularization all help; at evaluation, replay the same initial state under matched alternative commands and measure the state differences. Action can also be hierarchical — “take the right lane” is a goal, steering and torque are controls — and confusing the levels yields vague conditioning, where a language command nudges scene semantics without the temporal precision control needs.

5 · Visual fidelity and control fidelity are different axes

A polished video can fail as a world model in several independent ways, so separate the targets:

• Image fidelity: frames have natural texture, lighting, detail.
• Temporal fidelity: motion is smooth and appearance doesn’t flicker.
• State fidelity: objects keep identity, count, geometry, and hidden persistence over time.
• Action fidelity: changing an intervention gives the correct conditional consequence with correct timing.
• Physical fidelity: contact, collision, speed, conservation, and rare events obey the environment enough for the task.
• Decision fidelity: a policy chosen using generated rollouts actually succeeds in the real system.

These conflict. A pixel loss averages uncertain futures — blurry but reasonable expected motion. A perceptual or adversarial loss makes sharp frames, perhaps by sampling convincing but state-inconsistent detail. Diffusion can generate exquisite futures that barely respond to steering. A compact recurrent model may look crude yet track lane position and reward well enough to control. For the fork, compare predicted ego path, barrier clearance, braking distance, and other-agent tracks under left/straight/right — then look at appearance; human preference alone overweights texture and rarely notices that an occluded bicycle became a different bicycle, or that braking conjured the obstacle.

Named systems sit at different points on these axes. Image- and temporal-fidelity-first generators — Sora-class text-to-video, and NVIDIA's Cosmos 3 "Predict" for physical-AI futures — produce striking frames but were not built to obey a per-step control signal. Action-fidelity-first interactive models trade some polish for a controller that must respond every frame: Genie 3 generates real-time playable worlds from a prompt, GameNGen ran DOOM as a diffusion "game engine" under live controls, and DIAMOND trains a diffusion world model an agent can act inside. Driving-specific efforts (GAIA-class models, GenieDrive) push physical and decision fidelity by conditioning on 4D occupancy or maps. None dominates every axis — which is exactly why the six targets above must be scored separately rather than collapsed into one "looks real" number.

6 · Rolling generation needs a belief beyond the sliding window

Context is finite. A red car passes behind a truck and stays hidden for ten seconds; if that evidence scrolls out of the attention window, a pure next-window generator simply forgets the car, and on reappearance it may invent a new vehicle, change its color, or drop it. Bigger context delays the problem and raises attention cost — it does not define what should persist. So combine memory at several scales: recent high-resolution latents for exact near-term motion; a recurrent belief that summarizes history at fixed cost; object tracks or spatial memory that carry identities through occlusion; keyframes or retrieval memory to recover appearance when a place returns; and slow scene latents for layout, weather, and goals while fast latents handle motion.

Crucially, the memory update must distinguish observed from imagined evidence. During open-loop rollout, generated frames are samples from the model, not measurements — feed a sampled red-car position back as certainty and the model self-confirms its own hallucination. So the belief transition should grow uncertainty with horizon, and only a real camera frame should collapse or reweight hypotheses:

So long-horizon consistency ties straight back to lessons 2 and 6: video generation is an observation decoder around a belief process, not a replacement for belief.

7 · Multimodality needs whole futures, not indecisive averages

At the fork a pedestrian may wait or cross; predicting the average — half a pedestrian in the lane — is useless. A generative model should hold coherent hypotheses over the whole trajectory: one sample is “wait, then cross,” another “cross immediately,” and within each sample positions and reactions stay consistent frame to frame. Diffusion, flow, discrete-token sampling, mixture latents, or branching recurrent states can all carry alternatives — but diversity is not calibrated uncertainty. A model that invents ten cinematic outcomes while missing the rare collision is diverse and unsafe, so check that event frequencies and spatial distributions match data under controlled conditions, tails included. And for planning, share randomness across actions: generate paired rollouts from the same initial belief and exogenous seed so differences trace to the action, not to unrelated sampled weather — and sample enough hypotheses to estimate risk, knowing generative compute competes directly with decision latency.

8 · Objectives teach different — sometimes conflicting — behaviors

A video world model is usually trained with a weighted objective, and each term has a distinct job:

• Codec reconstruction/rate sets representable detail and compression cost.
• Generative likelihood / denoising / flow loss learns the conditional future in token or latent space.
• State probes supervise depth, segmentation, flow, objects, ego motion, contact, or reward so causal variables survive compression.
• Action-contrast loss penalizes rollouts that ignore or mis-time control; paired interventions are gold.
• Temporal/identity consistency rewards track persistence without forcing all futures identical.
• Task loss checks value, safety cost, or policy success so decision-relevant facts survive.

Two forces fight here. Teacher forcing trains on true past tokens while deployment conditions on generated ones — that exposure gap causes compounding error (lesson 06), and scheduled sampling, multi-step latent losses, block prediction, noise injection, and training on model-generated prefixes each patch part of it, but none replaces open-loop evaluation. And data composition matters as much as the loss: passive internet video teaches broad appearance but rarely actions or counterfactuals; robot logs give actions but are narrow and policy-biased; simulation gives controlled interventions and labels but a domain gap. Combine them deliberately — pretrain visual priors broadly, align to sensor/action logs, fine-tune dynamics on interventions — and balance common cruising against rare collisions so frequency doesn’t erase decision importance.

9 · Training and inference run on different clocks

Training processes many clips in parallel; control must sample futures within a deadline — so count end-to-end operations, not parameters. Autoregressive cost scales with generated token count even with a key/value cache; diffusion scales with denoising steps times block size; a recurrent latent transition rolls cheaply with only chosen trajectories decoded. That argues for a two-stage planner: roll thousands of candidates in compact latent state, score them with reward and constraint heads, then decode a few finalists for inspection or fine collision checks — decoding every candidate to video wastes compute if the decision only queries lane occupancy and risk. (A creative-media product inverts this and spends most compute on pixels, because appearance is the task.) Temporal stride adds one more knob: coarse steps extend horizon cheaply but skip fast events, so a hierarchical model can evolve slow scene state at low frequency and switch to a fine local model near contact — adaptive horizon and resolution beat uniform full-fidelity generation.

10 · Failure diagnosis with causal tests

Video errors are especially easy to misread, because a plausible decoder can hide a broken state. Walk the interfaces earliest to latest — codec → memory → transition → action alignment → planner — and stop at the first controlled expectation that fails.

1. Left and right commands make almost the same road. Same prefix and seed, change only the action; if the paths barely differ, conditioning is ignored or the data has too few action contrasts. Strengthen action injection and add counterfactual data — and pass only when the response is selective, not a wholesale scene change.
2. The brake response is a second late. The model may know braking physics but receive the control at the wrong token. Run an impulse-action calibration with known command, actuator, and frame timestamps, and locate whether the delay enters at control aggregation, the codec, or the actuator model. Fix the alignment — capacity can’t fix a shifted clock.
3. Frames stay sharp while vehicles change identity. A perceptual objective makes every local frame convincing without a persistent entity. Hide a tracked vehicle through a long occlusion and see which history the reappearing one inherits; if identity resets while texture stays sharp, add object/spatial memory and an identity loss. The diagnostic is track continuity, not human preference.
4. Rollouts blur at intersections. Several futures are possible, so deterministic regression averages them into a half-crossing pedestrian. Add a trajectory-level stochastic latent, generate whole branches, and score calibrated branch frequencies — sharp samples alone don’t prove the probabilities are right.
5. Rollouts are diverse but steering is random. Stochasticity beat averaging but now drowns control. Use the same exogenous noise for paired left/right rollouts; if unrelated scene changes dominate the action difference, regularize selective response and strengthen the causal control path. Useful uncertainty changes hidden events, it doesn’t erase the intervention.
6. Rare pedestrians disappear. Before blaming dynamics, reconstruct the observed input through the codec — if the pedestrian is already gone there, coarse stride or frequency-weighted training discarded the cue. Rebalance data, reduce stride, or add state supervision; only if the codec preserves the pedestrian is the transition the suspect.
7. The closed-loop policy exploits an impossible gap. The planner searches harder than the data policy and finds model error outside familiar actions. Run an adversarial planning audit against trusted execution, penalize epistemic uncertainty, correct with real data, and replan over a shorter validated horizon — success is exploitation vanishing without forbidding all useful novelty.
8. A long video starts repeating scenery. Recent frames look fine, but the window forgot where it is. Leave a place, travel past the context window, and return; if the scene resets or loops, add a map, keyframe retrieval, or slow scene state. Persistence is tested by return and loop closure, not by extending one forward clip.

Always test the representation before blaming dynamics: reconstruct held-out frames to test the decoder, probe latent state to test the transition, use paired interventions to test action causality, and only then test the closed loop — where a planner hunts precisely for the model’s weakest regions.

11 · Build an evaluation ladder, not one video score

Start at the codec: reconstruct held-out clips and measure both perceptual quality and decision probes — small-object recall, speed, signal state, contact timing, identity — stratified by object size and motion, so a lost brake light isn’t blamed on prediction. Next, one-step transition under observed context: report latent likelihood or denoising loss, but also decode state variables and calibrate their uncertainty. Then push open-loop horizon while withholding real future frames, plotting error, identity survival, collision recall, and calibration against simulated seconds (strides differ), separating static scenes, ego motion, object motion, interaction, occlusion, and rare events so an easy majority can’t hide the weakness.

The next rung is intervention: matched initial states with alternative actions and shared seeds, measuring a response curve — trajectory or speed as steering/brake magnitude varies — because correct sign is not enough; slope, delay, saturation, and collateral changes matter, and a zero-perturbation null test should not change unrelated attributes. Then distributions: for genuinely ambiguous intent, draw many rollouts and compare event frequencies, time-to-event, spatial coverage, and tail risk to held-out outcomes, using proper scoring rules (best-of-many hides a model that assigns tiny probability to the truth). The final rung is decision transfer: let the real planner choose actions through the model and execute in a trusted simulator, replay, or bounded real setting, comparing reward, violations, and regret against ground-truth dynamics and searching for exploitation — a model can have great average prediction and one exploitable blind spot the planner keeps finding. And report serving cost alongside accuracy (ms per observation, per denoising step, memory per environment, candidates before deadline), because that decides whether a nominally better model can participate in control at all.

12 · Product and system tradeoffs

Use case sets the fidelity. Entertainment wants diverse coherent pixels and controllable story semantics, not exact hidden-state calibration; robotics wants low latency, conservative uncertainty, geometry, and action response, with photorealism as a diagnostic; synthetic training data wants label consistency and coverage as much as realism; forecasting wants probability calibration and rare-event recall over one impressive sample. Serving follows: long-context transformers need cache proportional to history, diffusion rolls saturate accelerators, and token codecs save bandwidth but create a versioned interface — changing the codebook invalidates stored latents and checkpoints. Separate observed and generated caches, store action timestamps, and keep rollout seeds reproducible for debugging.

Safety adds guardrails: cap the planning horizon at where validation supports accuracy, replan on new observations, penalize epistemically uncertain regions, combine learned video with hard geometric constraints where possible, and monitor action sensitivity — the system should be able to say “this rollout looks plausible but is not trusted for braking distance.” And language conditions scene, goals, and high-level action but is context, not physics: a strong text prior can insert a stereotypical outcome that contradicts sensors, so ground every named entity to persistent visual state, keep instruction distinct from actuator command, and let new evidence correct the narrative.

13 · Bridge: video must become embodied belief

The delivery robot does not only see. It measures wheel speed, joint position, inertial motion, sound, touch, and sometimes language instructions. These streams arrive at different rates and expose variables video cannot identify. Lesson 13 first turns these passive video predictors into playable environments — action-conditioned, real-time, and memory-consistent enough to act inside — and Lesson 14 then ties the belief to a body: proprioception defines what the agent did, touch disambiguates contact, language specifies goals, and vision supplies broad scene evidence.

Interview prompts

• What does the video tokenizer decide? It sets the spatial and temporal resolution of representable events, the compute budget, and which decision-relevant cues are irreversibly discarded.
• Discrete tokens or continuous latents? Discrete tokens give categorical likelihood and explicit rate but quantize; continuous latents preserve smooth neighborhoods and suit diffusion/flow but are harder to density-model. Choose using codec probes and rollout cost.
• Why can a strong video generator be a weak world model? Image plausibility can coexist with identity drift, wrong causality, ignored actions, uncalibrated uncertainty, or poor closed-loop decisions.
• How do you test action fidelity? Hold initial observations and exogenous randomness fixed, vary only the action, then measure selective state changes, magnitude, and timing against real matched interventions.
• Why does rolling context forget the world? Long-occluded facts leave the finite window. Persistent recurrent, object, spatial, or retrieval memory is needed, with uncertainty that grows during imagination.
• How do teacher forcing and deployment differ? Training conditions on true prior tokens, while rollout conditions on model samples. Small errors become future inputs and accumulate; multi-step training and open-loop tests expose the gap.
• When would you prefer a compact recurrent model over photorealistic diffusion? When control needs many low-latency candidate rollouts and decision-state accuracy matters more than decoded texture. Decode only finalists if necessary.
• How should multimodal futures be evaluated? Check coherent trajectory samples, calibrated event frequencies including tails, state and action fidelity, and downstream policy transfer—not just diversity or human visual preference.