Interactive and playable worlds

A video model that only watches is a predictor. Put a live controller in the loop and demand that the movie stay a controllable, consistent place, and the same model becomes an environment—a neural game engine you can act inside.

The dependency

Lesson 12 built a video world model that predicts an action-conditioned future p(x_t+1:t+H | x_≤t, a_t:t+H−1) and evaluated it open loop: does the generated clip look plausible and respond to a supplied action? This lesson removes the supplied action. A person, policy, or planner now injects one action at a time and expects the world to answer immediately and remember what it showed. That single change—closing the loop with a live actor—turns three latent requirements into hard ones: real-time one-step rollout, controls that exist at all, and consistency that survives minutes of play.

Forced by 12An action-conditioned video model that predicts futures open loop. This stepClose the loop: a live controller supplies actions, so the movie must become a real-time, controllable, memory-consistent place. Forces 14A playable world is still bodiless; ground it in proprioception, touch, and language.

1 · The interface flip: from predictor to environment

The mathematics barely changes; the usage does. A predictor is asked, once, for a whole horizon under a fixed action plan. An environment is asked, repeatedly, for exactly one next frame under whatever action just arrived:

x_t+1 ∼ p_θ(x_t+1 | x_≤t, a_t), a_t = controller(x_≤t) supplied live at step t

Three consequences follow immediately, and each is a first-principles requirement rather than an engineering nicety.

Rollout must be autoregressive and real time. The next action depends on the frame the model just produced, so the horizon cannot be generated in one batched pass. The model must emit frame x_t+1, show it, accept a_t+1, and continue—inside a human-scale latency budget. A diffusion model that needs fifty denoising steps per frame, or a token model that decodes thousands of patches serially, must be distilled, cached, or made few-step to stay playable. Interactivity is a latency contract, not only a quality one.
Controls must exist and be disentangled. Open-loop prediction can succeed while ignoring the action, because the recorded continuation still looks right (lesson 12 §4). A player will not tolerate that: pressing left must reliably and repeatably steer the world left, holding everything else fixed. Controllability becomes the primary objective, not a diagnostic.
The world must persist under the player, not the data policy. A player revisits places, backtracks, and lingers—inducing a state distribution the passive video never contained. Objects that leave the frame must return unchanged; the scene must not melt into plausible-but-new content. Consistency over a long interactive horizon becomes the binding constraint.

Nothing here requires photorealism. A crisp, coherent, controllable low-resolution world is more useful as an environment than a beautiful clip that ignores the controller or forgets the room. The bar moved from looks real to behaves like a place.

2 · Where do the actions come from? Latent action models

The interface flip exposes a data problem. Internet-scale video—the reason lesson 12 could scale at all—has no action labels. Nobody recorded which button produced each frame transition. Yet controllability requires an action variable. The resolution is to infer the action from the video itself and learn a small vocabulary of controls without any labels.

Introduce a latent action. An inverse-dynamics encoder looks at a pair (or short window) of consecutive frames and asks what control best explains the change, then quantizes it to one of a few discrete codes:

ã_t = q( φ(x_t, x_t+1) ) ∈ {1, …, K}

Read it as an operation. The encoder φ compares before and after; the quantizer q snaps the difference to one of K latent actions. The world model is then trained to reconstruct the next frame given the previous frames and this inferred latent action:

maximize log p_θ(x_t+1 | x_≤t, ã_t), ã_t = q(φ(x_t, x_t+1))

The bottleneck does the work. Because ã_t must pass through a tiny discrete channel, the pair is pressured to route only the controllable, low-dimensional factor of the transition—move, turn, jump—into the code, while stable scene content flows through the frame context. At play time the encoder is discarded: a controller button is mapped to a latent code, and the same conditioned model now generates under a chosen action. This is the mechanism that lets a foundation world model become controllable from action-free video; it is the core idea behind Genie-style systems.

Latent actions are inferred, not given

Because the code is discovered, it need not match any human control scheme, and it inherits the coverage of the training video. A latent action that never occurred in the data—sprinting through a wall, reversing a one-way animation—has no reliable meaning. This is lesson 07's support requirement, now written into the control vocabulary itself: you can only steer with actions the video taught.

3 · Playability, defined

"Playable" sounds subjective; decompose it into three measurable properties, each traceable to an earlier lesson.

Controllability

Does the action move the world?

Changing only a_t from a fixed context must change the frame in a repeatable, disentangled way—lesson 07's intervention test, applied every frame.

Consistency

Does the world stay one place?

Objects persist through occlusion and return; geometry, count, and identity survive. Object permanence (lesson 10) over a long rollout (lesson 06).

Real-time

Does it answer in time?

Amortized one-step decode within a latency budget. Fidelity that misses the frame deadline is not interactive at all.

These trade against one another. Sharper frames cost latency; longer memory for consistency costs context and compute; stronger control conditioning can suppress rich background dynamics. A playable world model is a chosen point in that triangle, sized to the product—a research sandbox tolerates lower frame rate for fidelity; a game demands the deadline first.

4 · The consistency problem is compounding plus forgetting

Interactivity revives two failures we already derived, now coupled. First, autoregression feeds each generated frame back as input, so any per-step error compounds exactly as in lesson 06: over a long session the world can drift off its own manifold. Second, a player leaves a room and comes back, so the model must remember what is no longer on screen—the object-permanence requirement of lesson 10, but implemented inside a generative rollout rather than a structured belief.

The lever for both is memory. Whatever the architecture—a long context window, a recurrent state, an explicit spatial cache, or a retrieved keyframe—the world stays coherent only as far back as the model can condition. Write the interactive consistency as depending on that memory span M:

x_t+1 ∼ p_θ(x_t+1 | x_t−M+1:t, a_t)

Too small an M and the world forgets: an object drifts, changes color, or is regenerated fresh when it re-enters view, because the frame that established it fell out of context. Too large an M and cost grows—attention over the window is quadratic (lesson 15's accountant)—so real systems compress old context, keep a sparse spatial memory, or periodically re-anchor to a keyframe. The interactive horizon is not fixed by the model's fidelity; it is set by how far its memory reaches before drift or forgetting breaks the illusion of a place.

The widget makes the tradeoff physical. The action steers the landmark through the view—this is controllability. When the landmark leaves the frame, whether it is retained on the edge (remembered) or vanishes (forgotten) depends only on the memory window, and the playable horizon grows with it. Fidelity of each frame is irrelevant to both effects; a beautiful renderer with a one-frame memory still forgets the room the instant you look away.

5 · Worked example: a two-room world you can walk

Suppose each latent step advances the view by one tile, and a red vase sits in room A. A player walks from A into room B (four steps), waits, then walks back. With a memory window of two frames, the vase left context after two steps; on return, the model has no record of it and regenerates room A from scratch—perhaps empty, perhaps with a different object. The world was controllable (walking worked) but not consistent (the vase did not persist). With a memory window spanning the eight-step round trip, the vase re-appears in place: the frame that established it is still conditioning generation.

Now raise the stakes. Suppose per-step content error is small but systematic, biasing wall texture slightly warmer each frame. Over a four-step traverse the drift is invisible; over a four-hundred-step session the world has quietly become a different place—lesson 06's compounding, now experienced as a session that "rots." The two fixes are the same as before: shorten the unchecked interval (re-anchor to a keyframe, the interactive analogue of re-observation) or train the model to be stable and self-correcting under its own outputs. There is no separate "long video" trick; a playable world is a rollout, and rollouts obey e_k+1≈J·e_k+δ.

6 · Design families

The same interface admits several implementations. Each inherits the representation tradeoffs of lesson 12 and adds the interactive constraints above.

Family	How the frame is produced	Interactive character
Recurrent latent (classic)	Roll a compact latent forward, decode a frame; a controller acts in the latent "dream."	Cheap, fast, low-fidelity; the original "agent trained inside its own world model."
Autoregressive neural game engine	Condition next-frame generation on past frames and the action; trained to imitate a specific engine or game.	Real-time single-game fidelity; consistency bounded by context; can be distilled for speed.
Latent-action foundation model	Learn latent actions from unlabeled video, then generate controllable worlds across many scenes.	General and promptable; controls are discovered, so they are coarse and coverage-limited.
Interactive diffusion / flow	Few-step or streaming denoising per frame, conditioned on action and rolling context.	High fidelity and multimodal; the fight is latency and long-horizon stability.

The pixel-versus-latent representations, the token/diffusion/flow choices, and the diffusion mathematics itself are all developed elsewhere and are not repeated here. See lesson 12 for the visual latent and generative-future machinery, and the Generative Continuous track for diffusion and flow. What is new in this lesson is only the loop closure: live actions, latent controls, and consistency under a player.

7 · Evaluating a world you can act inside

Open-loop clip quality (lesson 12) and per-frame realism say almost nothing about playability. The relevant tests are closed-loop and interventional: hold context fixed and verify that each latent action produces the correct, repeatable change; walk away from an object and back and check that it returns unchanged; run long sessions and measure drift; and confirm the frame budget is met under real controller input. These are exactly the closed-loop and counterfactual world-model-trajectory evaluations developed in the Synthetic Vision Data track, which also situates modern interactive systems and their benchmarks. We adopt that evaluation contract rather than re-derive it; lesson 16 folds these interactive axes into the full capability vector.

Linear diagnostic

Ask, in order: does an action change anything (controllability), does the same action change the same thing every time (disentanglement), does the world survive leaving and returning (consistency), does it survive minutes (stability), and does it answer within the deadline (real time). A failure at any rung makes the rungs above it untestable.

8 · Failure modes and what they imply

Gorgeous frames, dead controls. The model ignores the action and continues the most likely video. Latent actions collapsed to a single code, or conditioning is too weak. Test intervention every frame; strengthen or re-quantize the action channel.
Controls work, world forgets. Objects reset when revisited. Memory window too short or no spatial cache. Extend context, add a keyframe/scene memory, or re-anchor periodically.
The session slowly rots. Textures, geometry, or count drift over hundreds of steps. Autoregressive compounding (lesson 06). Train stability under self-generated inputs; re-anchor to reduce the unchecked interval.
Actions bleed into content. Pressing a button also changes unrelated scenery. The latent action entangled control with appearance. Increase the bottleneck's disentanglement pressure; verify the code carries only the controllable factor.
Beautiful but unplayable. Per-frame generation misses the latency budget. Distill to few-step sampling, cache context, or lower resolution; interactivity is a deadline, not only a quality metric.
Off-distribution controls. A player attempts an action the video never showed and the world hallucinates. The latent-action vocabulary lacks support (lesson 07); constrain controls to the learned set or gather targeted data.

Where this points next

A playable world is a controllable, consistent, real-time environment—yet it is still bodiless. It generates observations, but it has no proprioception, no force or touch, no language goal, and no notion of the one physical agent acting through it. Lesson 14 grounds the model in a body: it fuses vision with joint state, contact, audio, and instructions on their real clocks, so that "what happens if I act" becomes "what will this body experience if it takes this action for this goal."

Takeaway

An interactive world model is a video world model with the action loop closed: the same p(x_t+1 | x_≤t, a_t), but generated one frame at a time under a live controller. That closure forces three properties—controllability (actions must move the world), consistency (objects must persist and the session must not drift), and real-time decode—and it forces a data answer: learn controls as latent actions inferred from unlabeled video. Playability is not extra fidelity; it is control plus memory under a deadline, and its horizon is set by how far the model can remember before compounding breaks the place.

Interview prompts

What changes when a video world model becomes interactive? The action is supplied live one step at a time, so the model must roll autoregressively in real time, actually respond to the control, and remain consistent under a player-induced state distribution.
What is a latent action model and why is it needed? Unlabeled video has no action labels, so an inverse-dynamics encoder infers a quantized latent action from consecutive frames; conditioning generation on it yields controllability learned from action-free data.
Why does the discrete bottleneck in a latent action matter? Forcing the action through a few codes pressures it to carry only the low-dimensional controllable factor, disentangling control from stable scene content.
Decompose "playability." Controllability (action changes the world repeatably), consistency (objects persist, no drift), and real-time decode within a latency budget—they trade against one another.
Why is long-horizon consistency the same problem as compounding error? A playable session is an autoregressive rollout, so per-step error follows e_k+1≈J·e_k+δ; re-anchoring to a keyframe is the interactive form of re-observation.
How does memory window set the interactive horizon? The world stays coherent only as far back as the model conditions; short memory forgets revisited objects, long memory costs quadratic attention, so systems compress or re-anchor.
Why can a photorealistic interactive model still be unplayable? If it ignores the control, forgets the room, or misses the frame deadline, fidelity is irrelevant—the bar is behaving like a place, not looking real.
How would you evaluate an interactive world model? Closed-loop and interventional tests: per-frame action effect and repeatability, object persistence across leave-and-return, long-session drift, and latency under live input—the trajectory-evaluation contract, not clip quality.