What is a world model?

Build the whole idea from one question an agent keeps asking: if I do this, what happens next? Everything else — a hidden state to track, an imagination to run it forward, and honest uncertainty — is forced by needing to answer that question well.

0 · A freeze-frame can't be acted on

Pause a video of a thrown ball. From that one frozen image, could you catch it? You can see where the ball is, but not where it is going — up, down, left, right all look the same in a single frame. The one thing you would need in order to act — what happens next — is exactly what the snapshot leaves out.

An agent lives inside that bind permanently. It never receives “the world.” It receives a stream of measurements, and it must choose actions whose consequences it cannot see until later. So before acting, it needs an inner engine that can answer: “if I do this, what is likely to happen — and does that help or hurt my goal?” That engine is a world model.

Read that question again, because it quietly lists its own ingredients. To answer “if I do this, what happens next,” four things must exist:

• something that persists through time — otherwise there is no “next” to speak of;
• some way to observe it, since the agent can’t read the world directly;
• an action whose consequences differ from another action’s — otherwise “if I do this” is empty;
• a question whose answer actually matters to the agent.

Remove persistence and there is nothing to track. Remove actions and you have a weather forecast, not a controller — it can tell you what will happen, not what you can make happen. Remove the question and the phrase “good model” has no meaning to attach to. Everything in this course is a consequence of taking those four ingredients seriously.

1 · Name the pieces of the loop

Step back and watch the world do two things behind the agent’s back, over and over: it changes, and it shows itself. Give each a name. At time t the world is in some physical condition s_t the agent cannot read. It leaks out a measurement o_t; the agent picks an action a_t; the world moves on to s_t+1. Written as the two hidden processes:

Four words, and each is exactly what the ball taught us:

• State s_t — just enough about the world to predict its next step without dragging along the entire past. For a rolling ball, position alone is not state (you can’t predict the next frame from it); position and velocity is.
• Transition T — how a state and an action fan out into next states. We write a distribution, not one answer, because contact, other people, and unmeasured details make the future genuinely uncertain.
• Observation model O — how a state turns into a measurement. A camera squashes 3D geometry through lens and lighting; a force sensor gives a different, noisy shadow of the same world.
• Action a_t — an intervention, not just another input. Its whole point is that holding everything else fixed and changing only the action should change the predicted future in the right way.

“State” never means every atom in the universe. It means enough for the questions and horizon at hand. A robot planning three seconds ahead may need poses, velocities, and what’s driveable; a cloth simulator needs local stretch and stress. Choosing the state is a modeling decision — but it may not drop a variable that changes the answer you promised to give.

Why the world is not its pixels

Here is the trap the freeze-frame set, stated plainly: the same picture can hide different worlds, and different pictures can hide the same world. A ball at one spot in the image could be heading left in one history and right in another — identical now, different next. And rotating the camera repaints nearly every pixel while the room hasn’t moved at all. Pixels mix what’s really out there with viewpoint, lighting, and noise. A useful internal state has to peel the lasting causes apart from their momentary appearance — at least whenever the task depends on the difference.

2 · The agent builds its own state

The true state s_t and true laws are off-limits — the agent only ever had the clues. So it does what a detective does: it keeps a running best guess assembled from everything it has seen and done. Call that guess x_t. Two abilities are the minimum:

Think of E as reading the clues into a belief — the encoder or estimator folds the whole history into one present summary. Think of F as pressing play in your head — the learned dynamics that advance that summary one step under an action. To actually answer something, a small readout G turns the imagined state into the quantity you care about — a future frame, a collision chance, a reward, whether the episode ends:

The hat marks imagined rather than freshly seen; k is how far ahead you look; the actions come from a planner or policy. Notice what is not in this contract: a pixel decoder. If the question is “will the gripper hit the shelf?”, painting a photorealistic frame is wasted effort — you needed a yes/no, not a picture.

That leads to the one rule your summary must obey. Call it predictive sufficiency: your state is allowed to forget anything that never changes an answer you promised — but never something that does. Concretely, if two different histories collapse to the same x_t, they must genuinely have the same future for every question and action you claim to handle. Merging a red box and a blue box is fine if you’re only navigating around them. Merging a parked box and one hurtling toward you is fatal the moment you promise collision-avoidance.

3 · Watch it work: braking for a hidden cart

Abstractions convince slowly; a number convinces fast. Put a warehouse robot in a straight aisle. A cart is 12 m ahead, crossing toward the robot’s path at 3 m/s; the robot rolls at 4 m/s. Each second it can coast or brake (−2 m/s² for one second). Let the state be the three numbers that decide everything: s_t = (d_t, v_r,t, v_c,t) — the gap, the robot’s speed, the cart’s closing speed.

One second of coasting closes the gap by how far each party travels: the robot 4 m, the cart 3 m.

Coast once more and another 7 m disappears — the two paths pass through each other inside the model: a crash before two seconds are up. Now brake instead. The robot slows from 4 to 2 m/s, so it covers the average, (4+2)/2 = 3 m; the cart still covers 3 m:

Brake a second time and the robot reaches 0 (covering ~1 m) while the cart covers 3 m — 2 m of daylight remains. Here is the whole lesson in one picture: the same camera crop of “cart, 12 m away” is consistent with the cart being parked or closing at 3 m/s. One image cannot tell them apart; a short history can. So the model must infer speed from successive frames, hold onto the cart while a shelf briefly hides it, and roll two different futures — one per action.

Now admit we’re unsure. Say perception gives 80% that the cart is closing at 3 m/s and 20% that it’s parked. Coasting crashes in the closing case, so its two-second collision chance is about 0.8; braking is safe either way. If a crash costs 100 and braking costs 3, then

Brake. And notice the trap you just avoided: “average” the cart to 2.4 m/s and you get a tidy single number that quietly erases the dangerous 80% branch. The average future is not a valid future. (Lesson 02 makes this belief explicit and Lesson 05 shows why averaging futures is a genuine mistake.) This tiny example already exercises the entire contract — infer hidden speed, persist through occlusion, feed the action into the transition, roll a future to answer a collision question, and let uncertainty change the decision. A system that only paints a plausible next frame of the aisle has done none of it.

While real observations keep arriving, they pull the internal state back onto the truth. Once they stop, imagination has to eat its own output: a small slip in position or speed becomes the input to the next step, and the error grows. The takeaway to carry forward: a longer rollout is not made trustworthy by making the video longer — trust has to be earned against outcomes, uncertainty, and the decision at the end. Lesson 06 measures exactly how that drift compounds.

4 · Where graphics, vision, simulation, and video actually sit

“Isn’t this just video prediction? Or a physics engine? Or 3D reconstruction?” The cleanest way to keep them straight is to ask which way the information flows. The Computer Graphics track runs world → picture: give it geometry, materials, lights, and a camera, and it synthesizes pixels (it calls this the forward problem). The Computer Vision track runs the hard way back, picture → world: from pixels, estimate labels, geometry, and motion. The 3D Vision track makes the camera and geometry in that inversion explicit.

A world model uses these but adds something none of them has on its own: time under your control. Rather than memorize a taxonomy, walk the loop once and watch each neighbor light up exactly one arrow — and go dark on the rest:

1. Start with a known scene and camera. A renderer goes forward to a picture. It answers “what would this look like?” — but it never asks what scene caused a real photo, nor how the scene would change.
2. Now hide the scene and keep only images. Static 3D reconstruction runs the inverse and may recover a beautiful, persistent room — yet it stays silent on what a push or a bump would do to that room.
3. Add time, but not your hand. A passive video predictor extends past frames into plausible future frames. It learns how things tend to move, which is not the same as what happens when you act differently.
4. Hand it exact state and an action. A physics simulator steps that explicit state forward — real dynamics — but usually assumes someone already turned raw camera or force readings into its clean variables.
5. Close the gaps. A world model starts from raw observation-and-action history, infers the present belief, and predicts action-conditioned futures. Even then the name is a promise, not a proof: whether its data and accuracy are good enough is an empirical question.

This is why you can combine these components but must not rename one of them. A simulator can be the dynamics inside a world model; a renderer can be its observation decoder; a video generator can produce an observation-level rollout. The mistake is to see one arrow succeed and declare the whole loop solved. Photorealism proves you can synthesize an observation. It says nothing, by itself, about object permanence, the effect of an action, or a calibrated collision probability.

5 · Four tests for whether the name is earned

A definition is only useful if it can reject an impressive fake. Run these four tests in order — each one needs the previous one to have passed, or there is nothing to test.

1. Persistence first. Hide an object behind something, or pan the camera until it leaves view. Does the model still believe it’s there, with roughly the right properties? If it forgets the instant the pixels vanish, it is running the toddler’s rule “out of sight, out of mind” — there is no stable state to roll forward, so stop here.
2. Then dynamics. Cut off observations and let it run forward on its own. Do error and uncertainty grow sensibly with horizon? Failing here reveals a frozen snapshot or a one-step parlor trick: it remembers, but it can’t evolve.
3. Then intervention. Freeze the situation, swap in a different feasible action, and compare against what that action really does. If the future barely moves, or moves the wrong way, the model learned “what usually happens next,” not “what my action causes.” A controller can’t trust it.
4. Finally, decision value. Hand it to a planner and measure whether task success, safety, or data-efficiency actually improve. This catches the sneakiest failure: pretty frames and low average error hiding a rare optimistic mistake sitting precisely where the planner will go looking.

The order is causal, not stylistic. No persistence, no hidden thing to roll forward. No dynamics, no trajectories to compare. No intervention, no answer to “what if I act differently?” And without decision value, none of the accuracy above proves the model is useful where mistakes actually cost something.

6 · Choose the world to fit the question

No model captures everything, so stop trying. The real design task is to pin down the question distribution: which things must be predicted, over which states, actions, and horizons? A warehouse controller asks about occupancy and collision one to five seconds out. A manipulation policy asks about contact, support, and grasp success. A video editor asks about appearance under camera motion. Same phrase, “world model,” three different machines.

That turns state design into a short, honest procedure:

1. List the decisions. What actions can the consumer actually take?
2. List the outcomes. What must be predicted — including rare safety events and how sure you are?
3. Set horizon and resolution. Centimeters over 200 ms of contact is a different world than lanes over 10 s of driving.
4. Keep distinctions that change an answer. If two histories need different futures under some action, never merge them.
5. Drop distinctions that never change an answer. This is where compression stops being lossy vandalism and becomes the point.

It’s also why “the model understands the world” is useless for engineering. Replace it with a contract you can test: “Given two seconds of camera, proprioception, and actions, predict a calibrated distribution over object occupancy and contact for feasible controls over the next three seconds.” Now data, architecture, loss, and metrics are all things you can argue about.

7 · Explicit, learned, or a bit of both

So the real fork isn’t “symbolic versus neural.” Ask, variable by variable: is it observable, is there a law I trust, is there enough data, and where must uncertainty be visible? A self-driving stack routinely mixes an explicit ego-pose and map, a learned detector, a probabilistic motion model, and a neural occupancy grid — that is one world model spread across representations. The other knobs (deterministic vs stochastic, objects vs a dense grid, metric 3D vs raw features, open-loop vs frequent correction, general vs task-specific) have no universal best setting either. The contract and the cost of failure decide.

8 · How a planner actually uses the model

The model comes alive inside a loop. Right now: fold the newest observation into the belief x_t. Propose a few action sequences. Roll each one forward and score the outcomes and their risk. Then — this is the part people skip — execute only the first step or two, look again, correct the belief, and repeat.

Read it in plain words: over a horizon H, pick the action sequence with the best expected total reward (the expectation averages the uncertain futures) while keeping something like collision probability under a limit. Whether the optimizer samples, follows gradients, or grows a search tree is a later detail; this information contract is fixed.

Why act for only a step or two and then re-plan? Because reality is a free error-corrector. If the model slightly misjudged the cart’s speed, re-planning at 10 Hz stops that mistake from compounding over two open-loop seconds. This is receding-horizon control: imagine far enough to see the consequence, commit briefly, then let the next real observation set you straight.

The loop also separates two failures people constantly confuse. Model error — the prediction is wrong. Planning error — the model is fine but the optimizer didn’t find the good action. Pit the planner against an oracle with true dynamics to tell them apart. And a genuinely counterintuitive warning: a stronger optimizer can make a weak model behave worse, because it searches harder for the model’s most flattering mistakes. (Lessons 08–09 return to this.)

Finally, compute forces a split. Painting pixels may help a human inspect a plan but is far too slow to score thousands of candidates. A practical system runs a compact latent transition for the wide search, a grounded head for scoring, and a renderer only on the two or three finalists. Let how often a prediction is queried decide how expensive it’s allowed to be.

9 · A model is only trustworthy where it has seen action

Here is the failure that sinks demos. A learned transition is reliable only near the state–action combinations it actually saw in training. Passive driving logs are full of competent actions; a planner will happily consider a hard swerve the data almost never contains. The model will still answer confidently — and that confidence is not evidence.

So watch the edge of the data on purpose: use ensemble disagreement or density as a smoke alarm, keep actions inside trusted envelopes, gather targeted intervention data, bake in the kinematics you trust, and fall back to a cautious controller when uncertainty spikes. And report coverage over state, action, and horizon together — a dataset can be visually diverse yet contain almost no action diversity, which is exactly what you need to learn what actions do.

10 · Six tempting beliefs that are wrong

• “Next-frame prediction is enough.” It can ride local optical flow, ignore the action, blur several futures together, and collapse after one step. Test intervention and long horizons.
• “Photorealism proves physical understanding.” Gorgeous textures can sit on top of wrong contact timing, wrong object counts, and broken permanence. Score the state and outcome variables directly.
• “A bigger latent is a more complete world.” Extra capacity often just memorizes appearance instead of velocity or uncertainty. Completeness is measured against your questions, not vector width.
• “One predicted future is fine.” Other agents and hidden variables make futures branch. A single rollout hides the very probability mass a risk-aware planner needs.
• “Low test loss means safe planning.” The planner actively hunts for the errors that look most rewarding. An average metric barely feels the rare optimistic mistake it will exploit.
• “More history automatically fixes partial observation.” History helps only if the estimator keeps the relevant evidence and the data actually reveal the hidden variable. No amount of video reveals the mass of a sealed, motionless box without poking it.

11 · How to reason about one out loud

Asked to design a world model, resist naming an architecture first. Walk from the requirements to the representation, in this order:

1. Interface: sensors, action space, prediction targets, horizon, latency, safety limits.
2. Hidden causes: velocity, occluded objects, friction, intent, camera pose — anything that makes the same observation mean two different futures.
3. State structure: dense grid, scene graph, object slots, sequence tokens, metric geometry, or a hybrid.
4. Uncertainty: Gaussians, discrete modes, ensembles, particles, or sampled latents — matched to the kind of ambiguity you expect.
5. Learning signals: future observations, features, rewards, occupancy, action/inverse prediction, consistency constraints.
6. Close the loop: compare action sequences, execute a short prefix, observe, update, re-plan.
7. Evaluate by slices: occlusion, camera motion, rare interactions, novel actions, growing horizon, calibration, and downstream regret.

A strong answer also states where the model stops being trustworthy. If the data are all passive, action effects can be confounded — the model sees “braking precedes stopping” without separating braking from the danger that made the driver brake. Fixing that needs interventions, exploration, simulation, or structural assumptions. Lesson 07 is entirely about this.

12 · Takeaway

One assumption is still hiding in plain sight: we kept saying “the belief,” as if the agent knows the current state. It does not. A single observation is usually consistent with many possible worlds. Lesson 02 takes that ambiguity seriously, turns it into a probability distribution over states, and derives the predict-then-correct update that keeps it honest as evidence arrives.

Interview prompts

• What is the minimal operational definition of a world model? A mechanism that infers internal state from history and predicts decision-relevant futures under candidate actions.
• Why is a current image usually not Markov state? It omits hidden causes such as velocity, occluded objects, and physical properties, so older evidence still changes the future distribution.
• Why is a renderer not automatically a world model? It implements state-to-observation synthesis but need not infer state or model action-conditioned transitions.
• Must a world model reconstruct pixels? No. It may predict latent state, occupancy, reward, collision, termination, or value if those outputs are sufficient for its query contract.
• How would you test action conditioning? Hold the same inferred context fixed, vary feasible actions, and compare the predicted differences with interventional outcomes.
• What is predictive sufficiency? Histories merged into one latent must have the same relevant future distributions under every action sequence the model promises to evaluate.
• Why can low average rollout error still harm a planner? The planner searches action space and may exploit a rare optimistic model error that average metrics barely weight.
• When is a hybrid model preferable? When some structure such as camera geometry or kinematics is reliable while residual contacts, appearance, or behavior are better learned from data.