Why one-step accuracy fails

A transition is never used once on a clean state. A planner composes it with itself hundreds of times, feeding each guess back as the next input. We’ll derive exactly how error snowballs through that loop — and build the training, evaluation, and control that survive it.

1 · The exam-with-the-answer-key problem

Here is the trap in one image. Train a model to predict one step and it can look flawless — because at every step, training hands it the real previous state, like a student solving each problem with the answer key open. Now take the key away and make it run the whole exam alone, each answer feeding the next. That is what a planner does, and it is a completely different test.

Make it precise. A dataset holds real-observation-inferred states z_0:T and actions a_0:T−1. One-step training minimizes

Every input z is a clean data state, and the target z′ is the next real one. This is teacher forcing: reality supplies the prefix every step, so a mistake at time t never poisons t+1 — training just resets to the truth. A free-running rollout is the opposite:

After step one, the inputs come from the model’s own distribution d_θ,k(ẑ), not from the data. So the transition is graded exactly where it was never taught. A slightly off-manifold output becomes the next input, which produces a worse output, which… This mismatch — clean histories in training, self-made histories in use — is exposure bias. Formally, small E_{d data}[d] does not imply small E_{d θ,k}[d]; you need the induced states to stay near data, the transition to be accurate in a neighborhood, or real observations to reset the error before it wanders off.

2 · Derive the snowball from a single step

Why does error grow the way it does? Take deterministic true dynamics z_k+1=F(z_k,a_k) and a learned ẑ_k+1=F̂(ẑ_k,a_k), and track the error e_k=ẑ_k−z_k. The trick is to add and subtract F̂(z_k,a_k) — the learned model run on the clean state:

Two clean pieces fall out. The first bracket is old error, amplified by how sensitive the model is to its input. The second is fresh error the model makes even on a perfect input — call it δ_k. Linearize the first bracket with the Jacobian J_k=∂F̂/∂z and you get the whole story in one line:

Read it like interest on a loan. J is the interest rate on your existing error, and δ is a fresh deposit of error each step. If J shrinks errors (contracts), they fade; if its dominant singular value exceeds one, they compound. Unrolling gives the accumulated total:

Every local slip is carried forward through all later steps. This is precisely why one-step MSE is not enough: two models with the same δ can have wildly different rollout error, because one has a calm Jacobian and the other an explosive one.

3 · Watch 3% become a different world

Numbers make the interest metaphor bite. Take one sensitive coordinate with a fixed amplification J=1.08 and a small steady bias δ=0.03, starting exactly right (e₀=0):

After one step the error is 0.03. After two, 1.08·0.03+0.03=0.0624. The closed form is a geometric sum:

At ten steps, 1.08¹⁰≈2.159, so e₁₀≈0.03·(1.159/0.08)=0.435. A validation report proudly saying “three-hundredths one-step error” hid a ten-step displacement fourteen times larger. And if a contact boundary sits just 0.4 units away, the imagined rollout crosses into the wrong regime — it thinks the object bounced when it really passed by — and from there the governing law changes discontinuously, so the true damage is worse than the linear estimate.

One might hope the bias averages out. It rarely does: learned errors aren’t random coin flips — the same unmodeled friction or timing offset repeats across the whole sequence, so systematic error accumulates coherently, step after step in the same direction. That is exactly the dangerous case.

Drag J below 1 and the same 3% bias settles to a bounded error δ/(1−J) — self-correcting dynamics. Nudge it above 1 and the identical bias explodes geometrically, soon crossing the wrong-contact line. The one-step error δ barely changed; the amplification decided everything. That is why “low one-step error” is not a promise about rollouts.

4 · Distributions snowball too, not just point errors

Stochastic dynamics turns a point error into a drift between whole trajectory distributions:

Because it’s a product, mistakes multiply. Miss a branch at step two and it gets zero descendants at steps three through H — no later sampling can bring it back. A slightly wrong mode probability compounds through every conditional branch. Too much variance scatters samples into unsupported states where the model is unreliable; too little gives confident, narrow rollouts that look stable but quietly drop valid outcomes.

Open-loop uncertainty usually widens with horizon — but not always. Sometimes a future event resolves it: two routes merge, or a stable attractor pulls states together. So don’t reward variance growth for its own sake; diagnose the shape of the predicted distribution, its event coverage, and its calibration at each horizon. For an RSSM, a sharp probe is the gap between the k-step prior and the posterior at the destination: the posterior saw the real future, the multi-step prior only had the start and the actions, so their divergence measures how much the free-running model failed to anticipate.

5 · Open loop, closed loop, receding horizon — three different products

“How accurate is the model?” is unanswerable until you say when reality is allowed to correct it. Start with one system and remove evidence in stages; each stage is a different product with a different requirement.

1. Teacher-forced one step — reality resets the state every step. The model gets a clean state, predicts one transition, and is snapped back to the truth. The requirement is accurate local dynamics near recorded states. The trap: reading repeated success here as “the simulator works.” Correction hides weak dynamics because the model never eats its own error. Diagnose by withholding the next correction and finding the horizon where error suddenly accelerates.
2. Open-loop rollout — reality gives the start, then vanishes. Simulation, plan scoring, and generated video work this way; every prediction becomes the next input. The requirement is stable self-composition plus calibrated branching. The failures are drift, lost modes, and variance that collapses or explodes. Diagnose with horizon curves from a common start, compared against the real continuation.
3. Closed-loop — reality returns after each action. A robot acts, sees, and corrects the prior with the posterior. Dynamics still matter (the prior must bridge sensor gaps and hold occluded objects), but now the pressure shifts to fast inference and robust correction; latency and prior–posterior mismatch dominate. Diagnose by logging the posterior innovation — how far new evidence moves belief — and checking big corrections line up with real surprises, not timestamp bugs.
4. Receding horizon — imagine far, act briefly, correct, repeat. MPC deliberately blends open-loop imagination with closed-loop evidence. It needs local accuracy over the planning horizon plus speed to replan before the world changes. Too short a horizon is short-sighted; too long or too slow can’t replan in time. Diagnose at a fixed wall-clock budget by varying horizon and execution chunk and measuring closed-loop cost.

So there’s no honest “this model is good at prediction” without naming mode and horizon. A weather model running months open-loop, a robot replanning at 20 Hz, and an anomaly detector corrected every frame face different information limits and need different stability, latency, and metrics.

6 · Train the composition you’ll actually run

The direct cure for exposure bias is obvious once you name it: stop only training with the answer key. Unroll the model during training — from a real inferred state, roll under recorded actions and score several steps ahead:

G reads a target from latent state (features, objects, reward, events); the weights w_k set how much far horizons count — uniform weights push long-run stability but can drown local learning, while discounted weights match control returns but may neglect rare late failures. Four complementary techniques build on this:

Latent overshooting skips decoding every imagined step. Infer posterior states along the real sequence, roll the prior from time t, and force its k-step distribution to match the later posterior:

The k-step prior p^(k) is repeated transition; the destination posterior is an evidence-informed target, and stopping its gradient stops both sides collapsing together just to shrink the KL. It trains the imagination path directly — though the posterior targets are themselves imperfect.

Scheduled sampling occasionally feeds the model’s own output as the previous input, with rising probability. It exposes mistakes, but it’s ambiguous: the recorded target came from the real prefix, while the sampled prefix may describe a different valid world, so punishing that continuation can drag a multimodal model back toward averages. Safe only when perturbations are small and the future stays semantically aligned.

State perturbation and recovery adds bounded noise around inferred states and trains the model to return to plausible trajectories — widening the supervised tube around the data without pretending a wildly wrong branch shares the original target. The perturbations must respect state geometry; random pixel noise teaches denoising, not dynamics.

Constraint and invariant losses penalize mass creation, identity switches, interpenetration, map violations, or impossible termination — restricting where errors can travel. Hard constraints stabilize but hurt when the rule is approximate; soft penalties expose the tradeoff and let data override the prior.

7 · Objective and architecture tradeoffs

Long-horizon pixel loss is dense but brutal: a one-frame phase shift makes every moving pixel wrong even when paths, collision risk, and reward are perfect — and the gradient nudges toward blur. Feature, object, event, reward, or occupancy losses are more forgiving and decision-aligned, but can miss detail a new task needs. Multi-scale systems often predict compact task state far out and render pixels only near-term.

Autoregressive models train naturally with teacher forcing (each true token is on hand), but free-running token errors cascade within a frame and across time; masked blocks, span corruption, or rollout fine-tuning add exposure at extra compute. Diffusion trajectory models predict a whole horizon jointly, sidestepping stepwise temporal exposure bias, but a fixed horizon and iterative sampling complicate long or interactive plans. RSSMs make recurrent rollout cheap and overshooting natural, yet their compressed state can hide slow drift.

Direct multi-step models map (z_t,a_t:t+H−1) straight to chosen horizons instead of applying one transition repeatedly — cutting compounding for a fixed query but losing a reusable one-step simulator and risking inconsistency across overlapping horizons. Hybrids keep a recurrent core plus direct event/value heads at longer timescales.

A horizon curriculum usually wins: learn solid one-step grounding, then lengthen unrolls as the induced state distribution becomes meaningful. Start with very long unrolls and you backprop noise through an untrained transition; stop at short unrolls and you keep the original exposure problem. Drive the schedule by validation curves — extend when current-horizon rollouts stay inside a calibrated reliability envelope — not by epoch count.

8 · A step-by-step recipe for reliable rollouts

1. Name the use mode and maximum useful horizon. Open loop, re-observe every step, or replan periodically? Choose state, event, and risk targets to match.
2. Get one-step inference right first. Verify action timing, posterior reconstruction, prior prediction, and calibration. Multi-step losses can’t fix a broken data pipeline.
3. Burn in a context. Feed several real observations to form (h_t,z_t); don’t score rollout before the state actually holds velocity and hidden context.
4. Cut the correction. For the rollout segment, use only prior samples and recorded actions — accidentally feeding future encodings quietly restores teacher forcing.
5. Score several representations. Combine latent overshooting with object, geometry, reward, continuation, event, and short-horizon perceptual losses — and log each by horizon.
6. Add small on-manifold perturbations. Train recovery from errors like the ones the model actually makes, refreshing the perturbation distribution as it changes.
7. Grow the horizon gradually. Manage memory with chunking or truncated backprop, but make sure dependencies longer than the truncation still get a training signal.
8. Stratify the hard transitions. Sample contacts, occlusions, mode switches, rare terminations, and sharp actions often enough to learn them — and correct the probability prior if you rebalance.
9. Fine-tune on policy-induced states carefully. Use real trajectories from the current planner or a safe exploratory policy; pure model-generated states have no guaranteed correct target.
10. Freeze an untouched rollout test. Evaluate horizons and action regimes you never used for curriculum decisions, including controlled shifts.

Memory grows with unroll length; checkpointing, mixed precision, latent-only heads, and random start/horizon sampling help. But never let a systems optimization sneak teacher forcing back in — for example, recomputing each future latent from the real frame to save memory.

9 · Evaluate curves and decisions, not one horizon

From one inferred belief, run two paths: a posterior-filtered path that gets each real future observation, and a prior-only path that gets only actions. The gap between them isolates transition drift from encoder/decoder error. Repeat across stochastic samples.

Then plot, at every horizon: latent/feature error; object position and identity; event precision, recall, and timing; reward and return error; constraint violations; NLL; interval coverage; branch-mode recall; and epistemic disagreement. Include the uncertainty–error correlation: an unsupported rollout should get uncertain before it becomes confidently wrong.

And test three kinds of action sequence. Dataset actions measure interpolation under the collection policy. Random or perturbed actions test local intervention coverage. Planner-selected actions expose model exploitation — the optimizer hunts precisely for the states where errors flatter it. A model that only shines on dataset actions is not ready to guide a new policy. Finally, measure downstream regret: let the model pick a plan, run it in a trusted simulator or the real system under safety controls, and compare achieved return against a stronger oracle. Prediction metrics are proxies; decision error is the product.

10 · Failure modes, read off the horizon curve

The horizon curve tells you when a rollout breaks; the shape and location of the break tell you why. Read each as a chain: symptom → mechanism → discriminating test → repair.

1. Good at step one, then a smooth drift. This is the e_k+1≈J_ke_k+δ_k signature — a systematic bias injected repeatedly, an expansive Jacobian, or both. Fit the observed error to that recurrence, inspect bias direction and local sensitivity, then add multi-step loss, train recovery from realistic perturbations, and fix missing action/timing inputs. Success flattens the growth curve, not just its first point.
2. Smooth until contact, then a sudden break. A contact boundary switches the regime (free motion → stick/slide/bounce/separate) and one smooth transition missed it. Align error plots to contact onset, not clock time; add a mode variable or contact head, oversample events, impose trusted constraints. Verify correct branch selection right after contact, not just lower error.
3. The rollout freezes. A conditional mean averages opposite motions to zero, or over-contractive dynamics pull everything to a static attractor. The video looks stable, so naive stability metrics reward it. Compare velocity/motion statistics, action responsiveness, and branch entropy against data; use a probabilistic objective, add velocity/task heads, reduce smoothing. Change one action from the same start — the world must move differently for the right causal reason.
4. The rollout goes noisy. Independent per-step noise makes identity and intent flicker, or the model inflates variance to hide bias. Track identity and invariants within each sample, compare predicted scale with actual residual by horizon; add a persistent scenario latent and coherent sampling, and use the likelihood’s scale penalty while improving the mean model.
5. Objects fade away. Under pixel loss, an object uncertain across several spots is “less wrong on average” if it goes transparent everywhere — averaging, not physics. Measure object count, existence probability, and permanence under occlusion; add object/occupancy heads, represent existence explicitly, use a multimodal density. A good fix keeps the object present within each branch even when branches disagree on where.
6. Overshooting KL falls while tasks worsen. Prior and posterior found a lazy truce — both collapsed to an uninformative code, so alignment improves because there’s nothing left to disagree about. Measure posterior information, decoder/task probes, and per-dimension KL; check stop-gradient placement, keep grounding losses, reserve an information budget. The joint criterion: later priors approach later posteriors and those posteriors still encode predictive facts.
7. Great on recorded actions, bad on planned ones. The collection policy never covered the optimizer’s chosen interventions, and the optimizer seeks exactly the flattering errors. Compare matched-start evaluations under dataset, perturbed, and optimized actions; if quality degrades in that order, add a support penalty, use a conservative planner, and collect targeted real data. Don’t manufacture labels by trusting unsupported rollouts.
8. Belief oscillates after every observation. The prior predicts one state, the posterior snaps to another, repeatedly — a prior–posterior mismatch, an unmodeled delay, or misaligned timestamps. Plot the innovation q−p against action and observation times; fix alignment first, then train correction dynamics or model the delay. A repaired system reacts to real surprises but doesn’t rhythmically twitch in routine motion.

So a rollout can be stable for the wrong reason — collapsing to an attractor. Pair stability with responsiveness and correct variation. The reliable move is always the same: isolate one physical regime, change one causal factor with the start belief fixed, and check that both the predicted state and its uncertainty move the way the real system does.

11 · Re-observation is part of the algorithm

A deployed agent needn’t hallucinate forever. It filters the newest observation into a posterior, imagines H steps, executes the first action or a short prefix, sees again, corrects, and replans. That MPC loop converts “simulate perfectly for a long time” into four tractable demands: accurate local transitions, calibrated uncertainty, fast batched imagination, and robust correction.

Replanning frequency trades model error against compute and smoothness. Short chunks cut drift but can be myopic or jittery; long chunks save compute and enable coordinated maneuvers but trust the model farther. Adaptive horizons help — plan farther in familiar, low-uncertainty regimes and shorten (or fall back) when disagreement rises. And correction isn’t free: sensors have latency, dropouts, and limits, so train with missing and delayed observations, or a system tested with perfect frame-by-frame correction but deployed with intermittent sensors was tested as a different product.

12 · Chaos sets a hard information horizon

Some systems are sensitive to initial conditions even with a perfect model. If initial uncertainty ε₀ grows roughly as ε(t)≈ε₀e^λt with positive Lyapunov exponent λ, the time until it reaches tolerance ε_max is

Scaling the network shrinks model error but cannot recover information the sensors never captured. Past this horizon, the right target changes from exact state to a distribution over coarse outcomes, conserved quantities, reachable sets, or risk. A weather model can lose a vortex’s exact position yet keep useful regional statistics; a manipulation model can lose pixel alignment yet keep “is it still graspable?” This is what stops the doomed chase for sharp long-horizon video — sometimes blur is honest uncertainty, sometimes a weak model, and calibration, added sensing, repeat experiments, and invariant metrics tell them apart.

13 · System-design implications

Attach a validity horizon to every rollout. The model service should report horizon-dependent uncertainty, support, and calibration — never an unqualified trajectory — and planners should reject queries beyond the validated envelope.

Optimize latency and accuracy together. A slower, more accurate model can be worse if it forces less frequent replanning. Compare closed-loop performance at fixed wall-clock and compute, not fixed imagined steps.

Keep posterior innovations observable. Log how far each new observation moves belief from the prior; growing innovations are an early warning of drift, distribution shift, timing faults, or broken sensors.

Separate fast and slow state. Fine steps for fast contact, coarse updates for slow intent or map context — multi-rate transitions cut long-horizon compute and stop thousands of updates rewriting stable context.

Protect against optimizer exploitation. Penalize uncertainty and distance from data, validate candidate plans with ensembles or higher-fidelity models, and keep hard safety constraints outside the learned reward — because more optimization can make an imperfect model behave worse.

Close the data loop. Store high-innovation states, planner disagreements, near-violations, and fallback activations; collect those regimes for real, then re-evaluate on a frozen test before widening the operating envelope.

14 · Bridge: accurate action effects require interventions

We can now measure and tame self-composition error under a given action sequence. But we quietly assumed the learned action→next-state relation is what the action causes. Logged data doesn’t guarantee that: a behavior policy chose actions from its observations, and hidden hazards shaped both the action and the outcome. A model can learn “braking predicts collision” because drivers brake when a crash is already likely — then wrongly imagine that braking causes the crash.

Lesson 07 separates passive association from intervention, and asks what action diversity, randomized exploration, causal structure, and counterfactual tests a transition needs before it can guide a planner instead of merely replaying the behavior policy’s correlations.

Interview prompts

• What is exposure bias in a world model? Training conditions on observation-inferred data states, while free rollout conditions on the model’s own imperfect states, creating an input-distribution shift.
• What does e_k+1≈J_ke_k+δ_k tell us? Existing error is transformed by local dynamics, while new one-step error is injected; expansive Jacobian directions amplify both over time.
• What is latent overshooting? Roll the prior several steps from an earlier state and match its destination distribution to the later observation-informed posterior.
• Why can scheduled sampling be biased in multimodal worlds? A sampled prefix may describe a valid world different from the recorded continuation, yet the method still forces it toward that original target.
• Why can long-horizon pixel loss be misleading? Small phase shifts dominate pixels and encourage averaging even when objects, events, rewards, and decisions remain correct.
• How does receding-horizon control reduce model burden? It imagines only far enough to choose a short action prefix, then uses a real observation to correct belief and replans.
• How would you test for model exploitation? Evaluate planner-optimized action sequences, compare with dataset and perturbed actions, and validate outcomes in a trusted simulator or real system under safety controls.
• What is an information horizon? The point beyond which initial uncertainty and intrinsic sensitivity make exact state prediction unresolvable; beyond it the model should predict distributions or coarser decision-relevant quantities.