Geometric world models

Put persistent state in world coordinates, separate sensor motion from environmental change, and use a differentiable sensor to connect that geometry back to observations.

1 · One ambiguity forces the whole split

Turn your head ten degrees and the entire room slides across your vision — yet nothing in the room moved. Now hold still while someone slides the mug: the mug slides across your vision too. Same pixel motion, opposite causes. Pixel displacement alone cannot tell “I moved” from “the world moved,” and untangling those two is the entire reason to store a world in coordinates instead of pixels. Factor the three mechanisms:

s_t is the environment in a stable coordinate system; c_t is the sensor’s pose and calibration; T changes the world under action and disturbance w; O renders that world from a viewpoint with noise η. Move only c and observations change while the environment does not; move the mug and s changes, so every camera should agree on its new world position. That split isn’t cosmetic — it buys the right invariance (“where is the mug in the kitchen?” shouldn’t change when the robot turns its head) and the right equivariance (if the robot’s frame moves by a known transform, stored positions transform predictably instead of being relearned).

2 · Geometry pays off because it makes queries direct

Why maintain 3D at all, rather than leave everything in a latent vector? Because of query cost. The robot keeps asking whether its gripper path hits the cupboard, how far the mug is from the counter edge, which surface occludes the mug, and what a new viewpoint would see. In an explicit metric state these are geometric operations; in an unconstrained embedding a learned net must approximate each one and can fail off-distribution. But geometry isn’t all knowledge — it captures spatial arrangement and often surface appearance, not mass, friction, fragility, motor delay, intent, or “this mug is Alice’s.” So a practical state pairs explicit spatial variables with learned hidden features:

The real question is never “should the whole model be geometric?” It is “which high-value constraints and queries should be guaranteed by the representation, and which stay learned?” Explicit coordinates can make collision and visibility reliable while latent features carry material and intent.

3 · A coordinate frame is a contract

Every position has to answer “relative to what?” A camera frame is handy for immediate perception but moves with the sensor; a robot base frame suits manipulation but shifts as the robot drives; a world/map frame supports persistence across motion but can drift when localization is uncertain; object-local coordinates are ideal for articulated shape. Treat transforms as typed interfaces: if ^WT_C maps camera coordinates to world and a point comes from depth as p_C, then p_W = ^WT_Cp_C. The notation matters less than the discipline: never combine states unless their frames and timestamps are known. Many apparent “model failures” are really stale poses, a flipped transform direction, or asynchronous sensors.

And localization uncertainty must flow into the world state. If camera pose is uncertain, fusing measurements as if exact blurs static surfaces or makes them look dynamic. A probabilistic system holds a joint belief over map, objects, and pose:

So sensing can correct both the world and the sensor pose. A cupboard edge that looks shifted might mean the cupboard moved, the pose drifted, or the depth was noisy — and the posterior arbitrates using priors and repeated observations.

4 · Pick a 3D state by the operation it must support

Work backward from the dominant physical operation. Each representation below answers one bottleneck well and another badly:

1. If the arm asks “is this volume safe?” thousands of times, use occupancy or a signed-distance field. Repeatedly decoding a latent to relearn collision is the wrong shape; a grid makes collision, free space, and ray traversal native. The cost: memory grows cubically with extent unless sparse, and fine surfaces plus named identity are awkward. Diagnose by shrinking cell size and watching both memory and thin-object collision recall.
2. If most motion is on the ground, collapse a dimension. Driving queries ground-plane motion, so a bird’s-eye-view or height map makes multi-camera fusion and path planning fast. Use it for every 3D task, though, and overhangs and stacked objects alias — distinct heights land in the same planar cell. BEV is an efficient navigation state, a lossy manipulation state; test it on height-separated obstacles.
3. If control needs surfaces, normals, and joints, expose them. A mesh or articulated model gives contact, normals, and deformation handles directly — editable and controllable when structure is known. It breaks when topology changes: a torn bag can’t be fixed by moving existing vertices. Diagnose with topology-changing events and keep a residual or remeshing path.
4. If the state must absorb measurements incrementally and render fast, keep explicit primitives. Points, surfels, and 3D Gaussians fuse local observations and splat into views efficiently — strong sensing substrates. But rendering primitives don’t automatically give persistent physical parts: watertight collision is hard and primitives split, merge, or drift. So they need a lifecycle and correspondence rule, and long-rollout identity — not one-frame render quality — is the diagnostic.
5. If arbitrary continuous queries and compact fidelity dominate, learn a field. An implicit neural field maps a coordinate to density, color, or distance without storing every voxel — avoiding a grid’s uniform memory. But planning may need many sequential network evaluations and online updates are costly, so query/update cost can dominate the control deadline. Measure queries per candidate plan, not model size or PSNR.
6. If actions target named rigid things, factor pose, shape, and latent properties. A mug keeps a stable identity and shape while a 6-DoF pose evolves, making rigid motion compact and compositional. Force the whole world into that template and fluids, clutter, and novel topology violate it — a residual dense state is required, and unexplained sensor error is the sign the object bias has become a mismatch.

For the kitchen robot the answer is hybrid: a coarse signed-distance field for arm collision, an articulated robot model, explicit mug and bowl poses, and a dense appearance map for view prediction — reject most actions with the cheap field, then use an accurate mesh near contact, because not every query needs millimeter precision. And any comparison must use common axes: a static field can be optimized for minutes, a world model may need a fresh state at 20 Hz. The derivations above explain the trades; the table just makes them quick to retrieve:

5 · From 3D snapshots to 4D identity

A separate 3D reconstruction at each time step is not yet a coherent 4D world. Rebuild the mug at t and again at t+1 and you still don’t know which surface point or entity persisted — and prediction under action needs exactly that correspondence. Several factorizations supply it:

• Rigid entities: store a shape in object coordinates and evolve a 6-DoF pose; every surface point inherits the transform.
• Articulated models: store parts and joints; evolve joint angles and root pose, so a cupboard door rotates instead of becoming a new surface each frame.
• Canonical space plus deformation: map a persistent canonical point to its time-dependent position — good for cloth and bodies if the deformation stays plausible.
• Scene flow or persistent primitives: predict motion for observed points or Gaussians — flexible, but identities can drift, split, or merge.
• Eulerian fields: represent quantities at fixed locations, right for fluids, where identity belongs to transported material, not grid cells.

The mug is well served by rigid pose plus shape; the person’s shirt needs deformation; steam over the sink is a field. Force one representation across all three and you either waste compute or encode the wrong persistence assumption. And a warning about a common shortcut: bolting time onto a radiance field, F(x,y,z,t) → (σ,c), can replay a recorded performance — “at second 3 the mug is here” — without any transition that accepts a push. Clock-conditioned interpolation answers what was recorded; only dynamics answers what would happen if we acted differently.

6 · Differentiable sensing closes the learning loop

Explicit 3D labels are expensive, so let ordinary sensors supervise hidden geometry through a differentiable observation model. Given state s and camera c, it predicts an image, depth, silhouette, flow, or contact; the residual against the real measurement sends gradients back into geometry, appearance, pose, and sometimes dynamics:

For a ray camera the sensor accumulates color and opacity along a ray; for a splat renderer, projected primitives contribute by depth and footprint; for a mesh, rasterization picks visible triangles. The exact renderer belongs to 3D vision — here its role is an observation likelihood: “if this were the state and camera, how probable is the measurement?” Multiple sensor heads pin down different ambiguities: RGB gives texture but confounds lighting, depth constrains visible range, flow constrains projected motion but mixes object with egomotion, silhouettes constrain occupancy without interior depth, touch is sparse but nails contact. Predicting several modalities makes it harder for a latent to hide a physically inconsistent shortcut. But differentiability is not identifiability: a wrong camera pose can be masked by warped geometry, lighting baked into texture, a moving object explained as view-dependent appearance. Geometry priors, calibration, temporal correspondence, loop closure, and intervention data are what select the intended explanation.

7 · Geometry lets the agent choose where to look

An observation model isn’t only for explaining images you already have — it can predict what a candidate sensor motion would reveal. The robot can’t tell whether the mug sits right behind the bowl or farther back, because the bowl occludes its base; a small sideways head motion creates parallax and different predicted silhouettes, so the robot can take that view before grasping. That is active perception. For a candidate camera action a^c, predict what each hypothesis would produce and prefer the view expected to cut decision-relevant uncertainty:

The entropy difference is expected information gain, though in practice you target only the uncertainty that affects the grasp — mug depth, handle visibility, clearance — since a grand scenic view can carry more bits yet be less useful. And active sensing exposes whether the model truly separated camera from world: if it predicts that turning its head physically moves the mug, it will pick nonsensical sensing actions; if its renderer is accurate but its uncertainty never updates, it will stare at the same uninformative surface. Evaluate both the predicted observation and the posterior contraction.

8 · Separate egomotion, object motion, and apparent motion

Dynamic scenes mix motions. A background point has nonzero optical flow when the camera moves though its world velocity is zero; a car can be static in the world yet move relative to an accelerating ego vehicle; a spinning object shows different velocities across its surface. Make the decomposition explicit enough for the task:

The first transform is world-to-camera egomotion, the second is object-to-world motion, p_O is a persistent object-local point, and the projection Jacobian J_π turns 3D motion into image motion. So image flow is an observation of combined causes, not the world dynamics itself. A common pipeline estimates ego pose from static regions, then assigns residual motion to dynamic entities — but that breaks when most of the view moves, an object dominates the frame, or the “static” ground is an escalator, so joint inference over pose and entity motion is safer. This distinction is decisive for forecasting: if an oncoming car appears to expand only because the ego robot rolls forward, a camera-frame extrapolation double-counts motion after braking, while world-frame velocity plus the planned ego trajectory gives the correct relative observation. Camera-only, object-only, and both-moving sequences should be basic unit tests.

9 · Dynamics should keep constraints while admitting hidden variables

Once the state is geometric, a naïve network can still predict impossible geometry, so useful parameterizations bake in structure: rotations stay on the rotation group, occupancy stays bounded, joints respect limits, and contact candidates only arise near surfaces — with residual learned dynamics for friction, compliance, and the rest. But the mug’s future after a push depends on mass, friction, liquid content, and whether the counter is wet — properties often invisible. Store them as uncertain latents h, update them from observed response, and predict a distribution:

A planner should then weigh several plausible slides instead of trusting a mean trajectory that passes through no actual future. Geometry says where interaction can happen; hidden physical belief says how it may unfold. Constraint losses help but shouldn’t become theater — penalizing predicted-mesh penetration is useful; declaring a rollout “physical” because the motion is smooth is not. Validate with intervention outcomes: contact timing, displacement under controlled force, conservation where it applies, and closed-loop task success.

10 · Training: separate what each signal teaches

A geometric world model usually mixes static scene data, passive video, and action trajectories, and each constrains different parts:

• Multiview reconstruction teaches the sensor model and spatial consistency, especially with known pose.
• Temporal photometric/depth loss teaches correspondence and motion, but can confuse lighting or occlusion with geometry.
• Pose, flow, or scene-flow supervision anchors coordinates where labels exist.
• Action-conditioned prediction trains the transition and separates intervention from passive correlation.
• Geometric regularizers encourage valid occupancy, smooth surfaces, rigid/articulated structure, and nonpenetration.
• Task losses test whether collision, visibility, grasp, or navigation queries are actually accurate.

Then split the tests three ways: held-out cameras test the sensor model O, held-out times test correspondence, held-out actions test the transition T. This stops one subsystem from masking another — a perfect renderer with wrong dynamics fails action prediction, strong dynamics with a miscalibrated sensor fails image alignment, and both can look fine on training trajectories. Schedules usually bootstrap: calibrate and learn an initial geometry/renderer, add temporal state, then learn action-conditioned dynamics and jointly fine-tune. Fully-joint-from-scratch is possible but lets too many errors compensate for one another, so keep diagnostic heads for pose, depth, visibility, and object motion even if the final planner uses a compact latent.

11 · Failure diagnosis by controlled contrasts

Geometry makes failures measurable, but one symptom can have several causes — so don’t guess from the final image. Hold two mechanisms fixed, vary the third, and inspect the earliest violated contract.

1. A static cupboard swims as the camera moves. Since it’s physically fixed, the error must enter through pose, depth, or a non-equivariant representation. On a calibrated static sequence, first freeze geometry and optimize only pose, then freeze pose and optimize only geometry; whichever removes the apparent motion names the culprit. The repair belongs there, not in dynamics.
2. The mug leaves a duplicate after it moves. The mapper fused the mug into the static world and created a dynamic track. Mask the tracked mug during map fusion and replay the move; if the duplicate vanishes and vacated cells clear, dynamic/static separation — not rendering — was the mechanism, so use layered state with explicit clearing.
3. Novel views look good, but a push is wrong. Good view synthesis only shows the observation model renders a fixed state; it says nothing about the transition. Reset to the same state, match the camera, apply different controlled pushes; if images stay plausible while displacement ignores the action, train and evaluate T separately from O.
4. Rollout geometry slowly melts. One-step pixels hide primitive-correspondence drift. Pick canonical surface points and distances that should stay rigid and plot their error with horizon; a systematic change means you need persistent correspondence or a rigidity-preserving parameterization, not a stronger perceptual loss.
5. Only RGB fails under new lighting. Before touching geometry, compare predicted depth and silhouette; if they hold while color changes, the shape is intact and the appearance model entangled illumination with texture. Condition lighting separately and keep geometry fixed during the test.
6. The collision checker misses thin handles. A coarse representation can’t answer a finer query however well trained. Group test objects by feature size and measure clearance error; if it rises below the grid or primitive resolution, add local fine geometry near contact rather than raising resolution everywhere.
7. Every sensor shifts together. A modality-specific decoder is unlikely to move RGB, depth, and touch the same way — the shared world state or pose is the common cause. Compare against independent localization or fixed landmarks, then correct the shared transform instead of recalibrating each head.

The general recipe: physical invariant → observed violation → competing mechanisms → controlled contrast → local repair. Orbit a static scene to isolate sensing; hold the camera and intervene on an object to isolate dynamics; replay one action from several viewpoints to test coordinate consistency; and probe geometry directly rather than inferring it from image quality alone.

12 · Product and system tradeoffs

Geometric state costs memory, bandwidth, and synchronization. Dense grids grow fast with volume; neural fields save storage but turn every collision or render into many queries; dynamic primitives render fast but need lifecycle management; object models make control cheap but need perception and a fallback for unmodeled matter. A real-time product separates rates: camera and proprioception arrive fast, local tracking and collision update at control frequency, global map and dense appearance update slowly, and the planner reads an immutable timestamped snapshot so geometry doesn’t change mid-query. Since rendering, rollout, and mapping fight for the same accelerator, a beautiful sensor model directly reduces how many action candidates you can explore.

Allocate accuracy by risk. A game generator tolerates subtly shifting geometry if frames are fun; AR needs stable surfaces under camera motion; a warehouse robot needs conservative free space and accurate contact near the gripper but not photoreal texture; a vehicle needs low-latency occupancy and calibrated long-range uncertainty. Different Pareto points, not one leaderboard. And treat coordinate/calibration contracts like production APIs — record frame names, transform provenance, timestamps, sensor versions, and uncertainty, because a silent calibration shift can corrupt months of learned dynamics; monitor reprojection residuals on static landmarks and action-response residuals, not just end-task success. Keep an escape path for out-of-distribution geometry: a rigid-object model meeting a hanging cable should not force it into a mug-like pose — track residual sensor error, allow an “unknown deformable region,” fall back to conservative occupancy, and gather a better view rather than invent a precise-but-wrong surface. Explicit geometry is valuable partly because its violations — penetration, reprojection error, unexplained depth — are measurable at runtime.

13 · Bridge: geometry and video solve complementary problems

Geometry builds in spatial consistency and makes planning queries explicit, but maintaining it is expensive and it doesn’t explain every visual or physical factor. Internet-scale video, meanwhile, offers abundant trajectories with no 3D labels. Lesson 12 studies models that compress and generate those trajectories directly — and the key comparison won’t be “explicit versus learned,” but whether the resulting state preserves identity, responds correctly to action, represents multiple futures, and supports the decisions we need.

Interview prompts

• Why separate camera pose from world state? Because sensor motion changes pixels without changing the environment. The split creates a stable belief and allows one world state to predict observations from many viewpoints.
• When does a 3D reconstruction become a world model? When it persists through time, updates from observations, advances under actions, represents uncertainty, and supports prediction beyond replaying captured views.
• Why is a time-conditioned dynamic NeRF not automatically controllable? It can interpolate what occurred at each recorded time without learning a transition conditioned on alternative interventions.
• What does differentiable rendering provide? A trainable observation likelihood from hidden geometry, appearance, and sensor pose to measurements, allowing 2D sensor residuals to correct 3D state.
• How do you choose voxels, fields, primitives, or objects? Compare native query cost, update rate, memory, required precision, and representation mismatch. Collision favors grids or meshes; view synthesis favors fields or primitives; entity control favors objects.
• What is the main challenge in going from 3D to 4D? Correspondence: deciding what persisted and how it moved, deformed, appeared, disappeared, split, or merged.
• What important variables does geometry omit? Mass, friction, compliance, intent, affordance, ownership, and other hidden physical or semantic factors. Hybrid latent state and uncertainty are still required.
• How would you distinguish renderer failure from dynamics failure? Test held-out views of a fixed state for the renderer, and matched initial states with controlled alternative actions for dynamics; probe depth and state trajectories separately from RGB.