Planning in latent worlds

Planning asks the model a bounded question right now: which candidate action sequence reaches the goal with acceptable risk? Search depth, rollout width, model bias, and real-time latency all collapse into one budget.

1 · Planning is optimization wrapped around prediction

The model already answers a conditional question: given belief b_t and a proposed action sequence a_t:t+H−1, what future states and outcomes follow? A planner simply wraps an optimizer around that answer — pick the sequence with the best predicted utility, subject to the action, state, and compute limits you face:

Term by term: H is the planning horizon; A is the set of allowed sequences (with bounds like steering rate or joint torque); c(s,a) is immediate cost (negative reward if you prefer to maximize); γ discounts distant cost; ŝ marks a model-predicted state; the expectation averages the uncertain futures. That last piece, c_terminal, estimates cost beyond the horizon — leave it out and the planner learns to shove every crash one step past where it can see.

Innocent as it looks, that objective hides four independent choices: how to represent current state, how to predict consequences, how to score goals and risk, and how to search actions under a deadline. “Use a world model for planning” is not an algorithm until you pin down all four.

2 · MPC turns feedback into error correction

You could optimize a whole sequence once and execute all of it — but then every model error rides along to the end. Model predictive control (MPC) instead uses a receding horizon: at each step it solves an H-step problem, executes just the first action (or a short safe prefix), takes a new observation, updates belief, shifts the window, and solves again.

Throwing away all but the first action isn’t waste — the rest were conditional promises about a predicted future, and once reality weighs in, revising them is the whole point. Replanning quietly converts feedback into model-error correction: a gust shoves the drone, a camera reveals the obstacle was closer than believed, the tires grip differently than the latent thought — and the next optimization simply starts from the corrected state. MPC doesn’t make model quality irrelevant (between corrections you still need accurate local effects, some failures outrun the control rate, and the optimizer can exploit error even in a short window) — but closed-loop correction lets a merely local model solve tasks that would defeat a long open-loop rollout.

3 · Random shooting: the honest baseline

What’s the simplest optimizer that could work? For continuous control, just guess a lot and keep the best. Sample N full action sequences from a proposal, roll each through the model, score it, take the winner. No derivatives, and it parallelizes beautifully:

It’s a great sanity baseline because its failure is legible. If good trajectories occupy a decent slice of proposal space, enough samples hit one. But with d-dimensional actions over horizon H, the search lives in roughly dH dimensions, and the useful fraction can shrink exponentially with horizon. Uniform motor noise also produces jittery, physically implausible controls. Smooth correlated noise, action-repeat, splines, a model-free policy proposal, or a shifted previous plan concentrate samples on realistic sequences. And a discipline that never expires: never compare optimizers without equalizing model evaluations and wall-clock latency — a method that looks better may just be running ten times more simulated transitions. Report the objective and what it cost.

4 · CEM: learn where to look, during this one decision

Shooting wastes samples in bad regions. The cross-entropy method (CEM) fixes that by adapting its proposal as it goes. Start with a Gaussian over the action sequence, sample candidates, keep the best K as “elites,” refit the mean and variance to those elites, and repeat for a few rounds. (The “cross-entropy” name is about fitting the proposal toward the elite distribution — it’s optimization, not the classification loss that shares the name.)

j indexes the CEM rounds; μ,σ² are the proposal parameters at every action time and dimension; the elites are the lowest-cost candidates; smoothing α ∈ [0,1) stops the distribution from collapsing too fast. Bounds are handled by truncation, clipping, or a tanh-squashed variable.

CEM needs no gradients, tolerates discontinuous collision penalties, and steers later samples toward promising regions. Its failure modes are just as concrete: it can converge to one mode and miss a narrow alternative, or lock confidently onto a model hole. Too large an elite fraction barely concentrates; too small makes updates noisy. A minimum variance, multiple proposal modes, restarts, and uncertainty-aware costs are the usual guards.

5 · Watch CEM steer around an obstacle

Put a planar point robot at (x,y)=(0,0) that must reach (4,0) in four actions, each a displacement of length at most one, with a circular obstacle of radius 0.7 at (2,0). Cost:

Squared goal error, plus 20 per predicted step inside the obstacle, plus a small control penalty. Start with a proposal aimed straight right: μ_0,k=(1,0), σ_0,k=(0.5,0.5). Straight-line samples collide, because the centerline goes through the obstacle.

Say round one draws 64 sequences and keeps the best eight — four passing above, four below. With a single Gaussian, their vertical components average back toward zero: a concrete multimodality failure. Suppose the top route wins the tie and the elite vertical means become (0.55, 0.45, −0.35, −0.45) — rise, cross above, descend. Smooth with α=0.2 and the new mean is 0.8×elite + 0.2×old-zero ≈ (0.44,0.36,−0.28,−0.36); round two samples that corridor and tightens.

Say the best sequence ends at (3.8,0.1), never enters the obstacle, and uses four unit actions — cost (−0.2)² + 0.1² + 0 + 0.1×4 = 0.45. The straight sequence reaches the goal but clips the obstacle once at p₂=(2,0) — cost 20 + 0.1×4 = 20.4. Crucially, CEM learned no permanent policy; it fit a temporary distribution for this one decision. MPC then executes only the first up-and-right action; if the robot moves less than predicted, the next observation changes p₁, the old sequence is shifted as a warm start, and CEM re-solves from the measured spot. That combination — temporary plan plus feedback — is why CEM-MPC shrugs off moderate model mismatch that would sink an open-loop four-step solution.

6 · Gradients and learned proposals add structure

If the dynamics and cost are differentiable, you can skip sampling: initialize the action variables and descend ∂J/∂a_t:t+H−1 through the unrolled model. In a high-dimensional action space this can use each model evaluation far more efficiently than sampling — but it’s vulnerable to local minima, saturated action bounds, chaotic long-horizon derivatives, and simply wrong model gradients. (A model can predict values fine while having derivatives that point the optimizer at nonsense.) A learned actor adds a different kind of structure: draw proposals near π(a|s), then let CEM or gradients refine them for the current goal. This splits habit from deliberation — the actor amortizes the common case, planning spends compute where this situation is unusual. Keep some broad samples anyway, or a biased actor can quietly stop search from ever finding a different route.

7 · Put risk and constraints inside the objective

Minimizing expected cost is wrong when a rare collision matters more than average progress. With a stochastic model you have options: optimize a worst-case ensemble member, a high cost quantile, or conditional value at risk:

At α=0.9, CVaR averages the worst ten percent of predicted costs. A simpler score is E[J] + κ·Std[J], with κ setting risk aversion. And keep the two uncertainties distinct: disagreement because the model lacks data argues for caution or exploration; irreducible outcome noise may demand a robust action even with abundant data. Above all, a hard constraint should not be a mere finite penalty if violation is unacceptable — because some predicted reward can always be made large enough to outweigh a finite number. Candidate rejection, control barrier functions, reachability filters, a verified emergency brake, or a separate safety shield belong outside the learned objective. Planning strengthens a safety case; it does not create one.

8 · Tree search for discrete, branching decisions

Shooting treats an action sequence as one flat vector. But in chess, board games, or routing, actions branch discretely and many prefixes are shared — so re-sampling whole sequences wastes work. Monte Carlo tree search (MCTS) stores those shared prefixes in a tree, and each simulation selects a child, expands a new node, evaluates its future, and backs up value along the path. Selection uses a PUCT-style score:

Q(s,a) is the mean backed-up value of action a; P(s,a) is a learned policy prior; N(s,a) is the visit count; c_puct tunes exploration. The first term exploits actions that look good so far; the second is large for promising-but-rarely-tried actions, steering simulations toward uncertainty within the tree (distinct from real-environment exploration).

Unlike independent shooting, MCTS pours repeated compute into promising prefixes and returns a refined action distribution at the root. It’s less natural for wide continuous action spaces (unless you discretize or progressively widen), and its irregular, sequential memory access is harder to batch on accelerators. Depth, simulation count, branching factor, and leaf-value accuracy jointly set quality.

9 · MuZero learns a model for search, not for reconstruction

Classical thinking says learn the actual next observation. MuZero asks a sharper question: what must a model preserve so that MCTS chooses good actions? The answer is reward, value, and policy-relevant transition structure — not necessarily pixels, colors, or a faithfully reconstructed board. It splits into three learned functions:

Representation h maps observation history to a root latent; dynamics g takes a latent and a hypothetical action and emits predicted immediate reward and next latent; prediction f emits a policy prior and a scalar value. MCTS uses g to expand hypothetical edges and f to prioritize and score nodes. Training unrolls recorded actions through g, supervising predicted reward by observed reward, predicted value by a bootstrapped or search-enhanced return target, and predicted policy by the MCTS root visit counts:

K is the training unroll depth; z is a value target (not a stochastic latent here); π is the search policy from visit counts. So the model is trained precisely so that search over its own latent predictions improves the decision targets that then retrain it — a self-consistent loop.

10 · Value equivalence is precise — and task-relative

A model is value equivalent for a policy class and reward family if planning in it gives the same relevant values as the real environment, even when its internal state is not a faithful reconstruction. Two physical states may collapse to one latent if no allowed action and target objective ever needs to tell them apart — and MuZero exploits exactly that freedom.

But this is not the universal claim that “only value matters.” Equivalence is relative to what you ask. Picture two visually identical boxes, one fragile, one reinforced: if the training reward only measures how fast the robot touches a box, material gets discarded — change the goal to “lift without damage” and the old latent can’t recover it. A chess model needn’t render wood grain, but it must keep every board distinction that changes legal moves or future value. So derive the representation from the job, not from a catalog of model families:

1. If the future job is unknown, preserve broadly predictive evidence. Observation prediction keeps features needed to predict sensors — improving reuse and interpretability. The naive hope that “accurate pixels ⇒ accurate decisions” fails both ways: capacity leaks into irrelevant texture, while a tiny reward-critical fact barely moves pixel error and gets dropped anyway.
2. If the job is stable and search is the bottleneck, preserve decision consequences directly. Reward, value, and policy prediction keep exactly what the current decision needs — compact and search-efficient, because wood grain and lighting never enter the latent. The cost arrives when the question changes: a new reward may need something compression threw away.
3. If it must decide now and transfer later, give the state several obligations. A multi-head model keeps current task signals plus selected physical or semantic facts — preserving focus while transferring better. The real cost: more objectives consume capacity, interact through gradients, and need deliberate loss balancing.

The diagnostic follows the requirement: probe the latent with counterfactual queries outside the training reward — change the goal, ask for a physical property, alter a constraint while holding observations fixed. If search fails because the needed distinction is simply absent even with a strong optimizer, the representation was sufficient for the old query and insufficient for the new one. “Minimal sufficient state” always needs you to finish the sentence: sufficient for what?

11 · What actually distinguishes the methods

The names click into place once you ask four questions in order: what object is optimized, when is it optimized, what action geometry does it assume, and what survives after this decision?

1. Random shooting optimizes by selecting among finite sampled sequences, at every decision, for any sampleable action type; afterward the samples vanish and only the world model (plus an optional proposal) persists.
2. CEM optimizes the proposal distribution, through several sample–elite–refit rounds per decision, best suited to bounded continuous control; the fitted Gaussian is temporary (discarded or shifted forward), and the persistent knowledge stays in the world model.
3. Gradient planning optimizes the action variables directly through differentiable dynamics, at each decision, for smooth continuous actions; what must persist is a differentiable model whose gradients are actually useful.
4. MPC optimizes no unique object of its own — it is the repeated plan/execute-a-prefix/observe/replan pattern, wrapping whatever inner solver you choose; the persistent system is estimator + model + objective + constraints + optimizer.
5. MCTS optimizes a selective discrete tree and its root visit distribution, at each decision, natural for shared-prefix discrete branching; the tree is mostly temporary, while model, policy prior, and value estimator persist.
6. MuZero trains the model that MCTS searches — learning representation, latent dynamics, reward, policy, and value across experience while MCTS runs online, usually over discrete actions; what persists is the value-equivalent latent model’s weights.

The prose above explains why each has its shape; the table is just a lookup once that’s understood:

This also gives a debugging order. If the chosen sequence is poor even inside a trusted model, fix the inner optimizer. If the first action is good open-loop but failures grow after disturbances, inspect state estimation and replanning frequency. If search is systematic but reality disagrees with its rankings, inspect the world model or objective. The method name alone never tells you which link broke.

12 · Diagnostics before you trust closed-loop success

1. Optimizer regret under a trusted simulator. On small cases where exhaustive or high-budget search is possible, compare the online optimizer’s cost with the reference — isolating search quality from model error.
2. Open-loop prediction by horizon. Replay recorded actions and measure state, reward, constraint, and uncertainty error at every depth. The planning horizon should not exceed where task-critical predictions stay meaningful without correction.
3. Predicted vs realized plan ranking. Execute safe candidate prefixes and check that lower predicted cost reliably means lower realized cost. Ranking usually matters more than pixel reconstruction.
4. Width/depth/latency curves. Sweep candidates, horizon, CEM iterations, or MCTS simulations, plotting task return and tail latency. A method that misses its control deadline is not better.
5. Warm-start ablation. Compare shifted previous plans, actor proposals, and cold starts — stale warm starts can trap an optimizer after abrupt goal changes.
6. Uncertainty and support trace. Plot ensemble disagreement or data distance along the selected plan versus random alternatives. Selection drifting toward high uncertainty signals exploitation.
7. Constraint stress tests. Perturb mass, friction, delay, obstacle position, and sensor noise; verify both planner behavior and the independent safety layer.
8. Search-model consistency for MuZero. Compare root values before search, after search, and against realized returns; inspect visit entropy and reward predictions by unroll depth. Strong search targets can’t compensate forever for drifting latent dynamics.

13 · Planning is a real-time systems problem

A controller with a 50 ms deadline has a finite inference budget. Roughly, model work scales with candidate count × horizon × stochastic particles × optimization iterations (batching and tree reuse change the constants). Longer horizon reveals distant goals but compounds error; more width covers alternatives but raises latency; more uncertainty samples estimate tail risk but cut throughput. Production design uses the whole stack: cache the encoded current state, batch rollouts, drop to lower precision where calibrated, warm-start from the shifted plan, spend extra search only when uncertainty or novelty is high, keep a small verified fallback controller, and record both mean and worst-case latency. If the model is a slow diffusion generator, it may make gorgeous video and still be unusable in a high-frequency servo loop — the representation has to match the control clock.

So choose abstraction by decision timescale. A millisecond stabilizer may use linear local dynamics; a one-second route planner uses latent objects; a minute-scale task planner uses symbolic subgoals. A hierarchical system plans coarse subgoals at long horizon and lets low-level MPC enforce smooth motion — one monolithic model need not answer every timescale at the same resolution.

14 · Failure modes and concrete responses

• Horizon myopia: the goal or a delayed hazard lies beyond H. Add a terminal value, hierarchical subgoals, or longer coarse planning — don’t just lengthen a fine rollout.
• Model exploitation: the optimizer finds an impossible high-reward path. Penalize epistemic uncertainty, restrict support, use ensembles and hard constraints, and collect corrective data near the proposed behavior.
• CEM mode collapse: variance vanishes around a mediocre corridor. Keep a variance floor, use restarts or mixture proposals, inject broad actor-independent samples, and retain elites carefully.
• Objective misspecification: the plan makes progress while creating jerk or near-misses. Add operational costs and constraints, then test for new loopholes — no optimizer can infer unstated preferences.
• MCTS branching explosion: too many actions get shallow evaluation. Improve policy priors, use action abstraction or progressive widening, and improve leaf values.
• Value-model non-transfer: a new goal needs latent facts absent from MuZero training. Retrain with new heads or use a richer multi-task state — search cannot recover discarded information.
• Feedback too slow: a dangerous event happens before MPC’s next correction. Raise the control rate, shorten action commitment, add a reactive shield, or push critical constraints into a faster inner controller.

Where this points next

Every planner here assumes its state keeps the entities and relations that make consequences predictable. A single unstructured vector can forget which object slid behind which wall, confuse camera motion with object motion, or lose identity across occlusion. Lesson 10 introduces object-centric state, spatial frames, and persistent memory so prediction and planning can factor by things, geometry, and relations.

Interview prompts

• Why does MPC execute only the first action of an optimized sequence? The remainder is conditional on a predicted future. After one action, a new observation can correct state and model error, so replanning from evidence is more reliable than honoring an outdated open-loop commitment.
• How exactly does CEM differ from random shooting? Random shooting samples once from a fixed proposal and selects the best sequence. CEM iterates: sample, retain elites, refit proposal mean and variance, and sample again, concentrating evaluations around promising regions.
• Is CEM an alternative to MPC? No. CEM is a trajectory optimizer; MPC is a receding-horizon control pattern. CEM is commonly used inside MPC to solve each finite-horizon problem.
• When would gradient planning beat CEM, and what could make it fail? It can be more evaluation-efficient in smooth, high-dimensional continuous systems with reliable derivatives. Local minima, saturated bounds, exploding long-horizon gradients, discontinuous costs, or wrong model gradients can make it fail.
• What does PUCT balance in MCTS? It balances backed-up action value against an exploration bonus derived from policy prior and visit counts. The bonus spends simulations on plausible actions that have not yet been tested deeply.
• What does value equivalence mean in MuZero? The learned model may differ from the physical observation dynamics yet preserve rewards, policies, and values needed by search for the specified task and policy/reward family. It is equivalence of decision consequences, not faithful reconstruction.
• Why can a value-equivalent model fail after a reward change? Training was allowed to discard distinctions irrelevant to the old objective. If the new reward depends on one of those distinctions, no amount of search over the old latent can reconstruct it.
• How do you separate a bad planner from a bad world model? Evaluate the optimizer against a trusted simulator or exhaustive small-case reference, and separately replay fixed real action sequences through the learned model. Search regret with an accurate model implicates optimization; incorrect rankings under fixed sequences implicate the model or objective.