Embodied multimodal worlds

An embodied model must bind words and pixels to one persistent world, fuse body state and touch with vision, and predict contact-rich consequences fast enough to close a control loop.

1 · Start from the decision, not the sensors

A robot can watch its own fingers close around a mug and still not know whether it’s holding it — the picture looks identical whether the grip is firm or already slipping. Vision cannot feel force. That’s the whole reason embodiment needs more than a camera, and the honest way in is a principle: a modality is useful only when it reduces uncertainty about a variable that changes a decision. More cameras are not automatically more intelligence. Begin with the action, its deadline, and its failure cost; then ask which hidden facts decide the consequence. The resulting belief should be a sufficient statistic — once the agent has it, raw history adds little to predicting task-relevant futures.

We’ll follow one product through the final three lessons: Countertop Assistant, a mobile manipulator told, “Put the red ceramic mug on the serving tray.” The mug may hide behind a plate, the counter may be wet, a person may cross the arm’s path, and the gripper has a small camera, joint encoders, wrist force/torque, tactile pads, and microphones. Success means the correct mug reaches the tray upright within 20 seconds — no spill, no collision, no excessive grip force.

State the decision and the hidden variables leap out: object identity and pose, but also mug contents, mass, friction, whether the fingers have real contact, human motion, the robot’s own joint configuration, actuator delay, and the meaning of “serving tray.” RGB reveals only some of these, so the model keeps a distribution over state, not a single visual scene:

where r is task progress and c is safety cost. That factorization is more than notation: sensing corrects belief, action changes the world, and the model must predict both the useful and the unsafe consequence. Concatenate sensor features and emit an action instead, and you may have a strong policy — but not an inspectable predictive world.

2 · Each modality resolves a different ambiguity

Don’t memorize a sensor catalog. Start from a hidden fact that could change the action, ask why current evidence can’t settle it, and only then reach for the next measurement. For Countertop Assistant that unfolds in five steps:

1. First establish the external scene with RGB and depth. At 15–60 Hz these expose appearance, geometry, and motion, cutting uncertainty about object identity, pose, free space, and where the human is. But an image of fingers around a mug still can’t prove force is being transferred.
2. Then locate the robot from inside its own body. Proprioception reports joint angle, velocity, and effort at 100–1000 Hz and resolves body configuration even when the arm is occluded or blurred — because the same camera image can match different joint configurations that permit different safe motions.
3. At contact, swap visual inference for local physical evidence. Force and touch measure contact, pressure, shear, and slip at 100–2000 Hz, cutting uncertainty about friction, compliance, grasp stability, and incipient collision. The high rate isn’t decorative: slip can become unrecoverable between two video frames.
4. Listen for events the camera can’t see. Audio carries impacts, motor sounds, speech, and distant events, sampled at 16–48 kHz then summarized — revealing something outside the field of view, or a material response like a dull plastic tap versus a sharp ceramic ring.
5. Finally, let language say what the physical evidence is for. Language gives goals, names, constraints, and procedures, event-driven rather than clocked. It resolves which outcome matters and which prior knowledge applies — but it can’t measure the mug’s pose or prove a grasp is stable.

These are complementary, not interchangeable: vision shows fingers around a mug while tactile shear reveals it’s sliding; encoders localize an occluded elbow; audio reports a ceramic impact off-camera; language picks which of two identical red vessels the user means but can’t measure its pose. Good fusion lets evidence compete — if vision predicts a grasp while touch reports no contact, belief in “grasped” must fall.

3 · Put every event on a physical timeline

Multimodal fusion is first a timing problem, and it’s where most silent bugs live. A packet’s arrival time is not when the world was measured: a camera integrates light over an exposure window, audio represents a window, a force sample is nearly instantaneous, and an action command passes through networking, a controller, and actuator dynamics before motion starts. Let τ be physical event time, tᵐ a modality clock, δᵐ a clock offset, and ℓᵐ transport/preprocess latency:

Calibrate, store, and sometimes estimate these offsets online. Interpolate continuous proprioception to a query time; keep an image’s exposure start and end; pool tactile events over a short causal window; carry language as goal context rather than re-injecting it as a fresh observation. And never use a future sensor sample when building a state for real-time deployment — that’s silent look-ahead leakage.

Work the timing numbers

Say the overhead camera runs at 30 Hz with 10 ms exposure and 35 ms processing delay; proprioception at 200 Hz after 4 ms; tactile at 1 kHz after 2 ms; commands at 50 Hz; and the gripper starts responding 18 ms after a command. A camera frame arrives at wall-clock 1.200 s. Its exposure midpoint actually describes physical time 1.200 − 0.035 − 0.005 = 1.160 s, while the freshest proprioceptive packet describes ~1.196 s. Naively concatenating them places the body 36 ms ahead of the image.

During a 0.8 m/s wrist motion, 36 ms is 0.8 × 0.036 = 0.0288 m — almost 2.9 cm, far beyond a few-millimeter grasp tolerance. The correct record queries joint state at 1.160 s, pairs the image with actions executed before then, and keeps newer tactile/proprioceptive data for the next correction. If the command left at 1.150 s but execution began at 1.168 s, the 1.160 s frame is evidence from before the intervention — label it post-action and you teach reversed causality. So test alignment with known impulses: flash an LED when commanding motion, tap the force sensor in view, correlate motor acceleration with encoder velocity, and plot cross-correlations per session. If the peak drifts, fix the data before touching the model.

4 · Ground language in persistent entities, not one-frame labels

“Put the red mug behind the plate on the tray” packs a referent, a spatial relation, a destination, and a desired transition. The model must bind “red mug” to an object track that survives occlusion, “behind” to a coordinate frame, “tray” to a receptacle with capacity, and “put” to a controlled contact sequence. Grounding is not a per-frame segmentation label; it is a durable correspondence between symbols and changing world state. A useful hierarchy separates four coupled layers:

1. Semantic task state: goal predicates like on(mug, tray), constraints, object roles.
2. Persistent scene state: object identity, 3D pose distributions, occupancy, articulated relations, human tracks.
3. Interaction state: grasp mode, contact graph, estimated mass/friction, slip probability, local geometry.
4. Body state: joints, velocities, controller mode, saturation, recent executed action.

Updates flow both ways: pixels and touch update object/contact state; semantic identity tells vision which occluded track deserves memory; the goal decides which prediction errors matter. But language must not overwrite physical evidence — if a user calls a steel cup “the ceramic mug,” ground the referent while keeping material belief uncertain. Spatial language also needs an explicit frame: “behind the plate” could mean farther from the user, farther from the robot, or behind in the plate’s own orientation. Store transforms among camera, gripper, base, map, object, and human frames with timestamp and covariance, and compose them instead of learning a separate opaque relation per camera — a camera-frame point becomes a base-frame point via pᵇ = Tᵇ꜀pᶜ, and depth and calibration uncertainty must propagate into the grasp distribution. Test it by moving the camera with the scene fixed: world-frame belief should stay put even as pixels change; if predicted actions rotate with the camera, the model confused image coordinates with physical ones.

5 · Fusion architecture is a choice about conditional dependence

Early fusion turns every modality into tokens under shared attention — rich cross-modal interaction, but high-rate signals dominate the token budget and missing sensors destabilize it. Late fusion builds a per-modality state and combines distributions — modular and debuggable, but it can miss fine timing like a tactile spike lining up with one image edge. Cross-attention fusion keeps specialized encoders and lets a belief query attend to timestamped modality memories — often the useful middle ground. From a probabilistic view, independent evidence would update as p(s|o¹,…,oᴹ) ∝ p(s)∏ₘp(oᵐ|s) — but real sensors aren’t conditionally independent (two cameras share lighting errors; force and motor current share actuator effects), so multiplying confident experts becomes overconfident. Learned fusion therefore needs correlated-noise training, sensor dropout, reliability indicators, and calibration by modality combination, and it must distinguish “no tactile contact” from “tactile sensor unavailable.”

For Countertop Assistant: vision/depth tokens update object and free-space slots at 15 Hz, proprioception updates body state at 200 Hz, tactile summaries update a contact latent at 500 Hz, and the language goal seeds semantic predicates, all consumed by a recurrent belief core as timestamped deltas rather than one enormous dense grid — so during a camera dropout, body and contact beliefs keep evolving while object-pose uncertainty expands. Train the combinations the product will actually face: random per-stream dropout is too easy, because real failures come in bursts correlated with the event (a gripper camera occluded exactly at contact), so simulate long outages, saturation, stale packets, calibration jumps, and mutually inconsistent sensors. Add an auxiliary objective that predicts one modality from the others, but don’t demand perfect cross-modal reconstruction — touch carries local friction that distant RGB genuinely cannot recover, so the right output is a broader posterior, not a hallucinated tactile signal. Evaluate every sensor subset as its own regime and define which subsets allow normal operation, reduced speed, or a mandatory stop.

6 · Match the hierarchy to physical bandwidth

No single model rate fits a body. Human intent and room layout change slowly; a reaching motion evolves over tenths of seconds; slip turns irreversible within milliseconds. Running one big model for everything fails both ways: a 2 Hz foundation model can’t safely close a 1 kHz torque loop, and a 1 kHz language-conditioned transformer would burn nearly all its compute re-deciding a goal that hasn’t changed. So build a ladder whose abstraction drops as its rate rises, and follow one command down it:

1. Task layer — decide the phase. At ~0.2–2 Hz over tens of seconds, pick “approach, grasp, carry, place” and choose recovery when a phase fails. Language, predicates, and object relations fit, because the question is semantic.
2. World-prediction layer — compare consequential futures. At ~5–20 Hz over 1–5 s, branch action chunks and compare their human and object futures. Objects, geometry, and a stochastic latent fit, because occlusion and contact uncertainty must sway the comparison.
3. Motion layer — turn the chosen future into a trackable target. At 50–200 Hz over 50–500 ms, track collision-free pose and force targets using robot state, a local signed-distance field, and contact mode, where small geometric deviations now outrank linguistic detail.
4. Reflex layer — prevent irreversible local failure. At 500–2000 Hz over milliseconds, verified local control limits torque, arrests slip, and triggers an emergency stop — it can’t wait for a new semantic plan, because its deadline is shorter than that plan’s inference time.

So information flowing down gets more specific — subgoal, then action chunk, then pose/force target, then actuator limit — while evidence flowing up reports execution, violation, and changed belief. A good stress test injects a disturbance at each timescale: a changed instruction should move the task layer, an approaching person the world/motion layer, a sudden tactile slip the reflex — before any new high-level plan exists. The layers exchange contracts, not raw thought: the slow model proposes a grasp pose, force range, expected contact time, and uncertainty; the fast controller reports execution and deviations; when measured force breaks the contract, the reflex acts now and the world model revises belief later. Action chunks cut planning cost, but an interrupt channel must stay open, or a 500 ms open-loop chunk will ignore a person stepping into the workspace.

7 · Contact makes hidden physics decision-critical

At lift time, two RGB images can be identical even though one grasp is safe and the other will slip — so the deciding variable isn’t in the pixels at all. A friction condition shows why. If the mug has mass m, two opposing fingers each supply normal force N, and the friction coefficient is μ, a quasi-static lift needs 2μN ≥ mg. For a 0.45 kg mug at μ = 0.35, minimum force per finger is N ≥ (0.45×9.81)/(2×0.35) = 6.31 N. Let water drop friction to 0.18 and the requirement jumps to 12.26 N — possibly above the safe pressure for a fragile mug.

Vision gives a prior over material and wetness; a small test lift updates it via tactile shear. Say belief puts 20% on low friction, and a micro-lift produces a slip signature that occurs with probability 0.8 under low friction but 0.1 under normal friction. Bayes’ rule:

One cheap observation lifts low-friction belief from 20% to 66.7%, and a rational agent now lowers acceleration, replans the contact, or asks for help rather than just squeezing harder — active perception again, choosing a low-cost action because its observation separates hypotheses. Contact prediction should carry discrete modes — free, touching, sticking, sliding — plus continuous state within each, because averaging across modes invents a physically meaningless “half contact.”

8 · A VLA policy and a world model solve different bottlenecks

A vision–language–action policy amortizes (observation, instruction) → action into one forward pass — fast, and good at human-like behavior learned from demonstrations. A world model predicts consequences under alternative actions, supporting planning, explanation, and goal changes. Neither wins everywhere:

So in our product the VLA proposes three familiar grasp trajectories, the world model branches them and predicts mug stability and human-collision probability, and one is selected under a risk constraint; the motion controller tracks it while a tactile reflex catches slip. Expensive prediction goes where alternatives matter, not at every motor tick.

9 · Simulation is a prior; deployment is continual belief correction

Simulation can vary mass, friction, texture, lighting, camera pose, delay, and wear, but it cannot enumerate the real world. Domain randomization broadens support, system identification estimates explicit parameters, and residual models learn the sim-to-real gap — and all three should feed one online procedure: detect mismatch, widen uncertainty, shorten the trusted horizon, take information-gathering actions, and fall back outside the calibrated envelope.

The Synthetic Vision Data track develops the complementary data pipeline: branch controlled engine interventions from matched states, render the deployed sensor chain rather than ideal RGB, and use real failure slices to decide what the simulator must vary next.

Keep diagnostics causal and layer-specific: replay the same executed action with sensors masked one at a time, plot prediction error against time offset, check identity survival through occlusion, compare predicted and actual contact time, and bucket calibration by material, speed, lighting, and embodiment revision. Read attention only as a clue, never proof of grounding — and above all, intervene: change grip force while holding the visual context similar and verify predicted slip moves in the correct direction.

10 · An embodied model should choose what to sense

When uncertainty blocks action, the agent needn’t wait — it can move the camera, rotate the object, tap a surface, close the fingers gently, ask a question, or pause for an occluder to pass. These are epistemic actions: low immediate reward, high value because their observations improve later decisions. Embodiment turns perception into control. The value of information is the best expected decision after an observation minus the best decision under current belief, minus sensing cost:

Suppose the robot puts 40% on the visible red object being the requested mug and 60% on a similar container. Grasping now yields +10 if right and −8 if wrong, so expected utility is 0.4×10 + 0.6×(−8) = −0.8. A 0.5 s viewpoint change costs 1 utility unit and reveals identity with 95% accuracy; afterward the robot acts only when the view says “mug,” giving approximately 0.4×0.95×10 + 0.6×0.05×(−8) = 3.56, and 2.56 after the sensing cost. The look is rational even though it makes no direct task progress. But information gain isn’t everything — a perfect close-up might require moving through a person’s space while a weaker side view is safe — so the model should predict both how an action changes physical state and which observation it yields, and active sensing should be scored by final task risk and efficiency, not entropy alone. Test, too, that the model doesn’t invent information: after a camera move that leaves the mug occluded, uncertainty should stay high rather than collapse because the action usually helps in training.

11 · Design checklist for an embodied model

1. State the decision, control deadline, horizon, and unacceptable outcomes.
2. List hidden variables that change consequences; map each to evidence and remaining ambiguity.
3. Define clocks, exposure windows, action execution semantics, coordinate frames, and calibration versions.
4. Choose persistent semantic, spatial, interaction, and body state; specify how missingness changes uncertainty.
5. Assign each prediction/control responsibility to a physical rate and an interrupt path.
6. Train with cross-modal contradictions, dropouts, delays, interventions, and rare contact transitions.
7. Measure modality ablations, timing sensitivity, causal action effects, uncertainty calibration, and closed-loop recovery.

Where this points next

Embodied intelligence turns one episode into a dense, asynchronous trajectory: multiple cameras, high-rate proprioception and touch, actions, language, calibration, and safety labels. Resolution × views × frame rate × context already produces many tokens; planning multiplies inference by candidates, horizon, refinement, and uncertainty samples. Lesson 15 follows this exact Countertop Assistant record through data curation, tokenization, distributed training, rollout accounting, and a latency-constrained service.

Interview prompts

• Why start from the decision instead of listing sensors? The required action, horizon, deadline, and failure cost determine which hidden variables matter; sensors are valuable only insofar as they reduce uncertainty about those variables.
• What ambiguity does proprioception remove? It reveals body configuration and motion when external vision is occluded, delayed, or geometrically ambiguous, and distinguishes environmental motion from self-motion.
• Why can timestamp alignment be more important than model size? A 36 ms mismatch at 0.8 m/s creates almost 3 cm of false geometry and can reverse action–effect order, so a larger model simply fits corrupted causality.
• What makes language grounding persistent? A phrase binds to an object track, relation, coordinate frame, and goal predicate that survive occlusion and are revised by later physical evidence.
• When would you prefer cross-attention over early fusion? When modalities have different rates and specialized structure, and you want a shared belief to query timestamped memories without making high-rate signals dominate one token stream.
• Why use a control hierarchy? Semantic planning changes slowly while contact reflexes need millisecond response; layered contracts preserve rich reasoning without missing physical deadlines.
• How can a VLA and world model cooperate? The VLA cheaply proposes in-distribution action chunks, the world model predicts and compares their consequences, and fast verified controllers enforce local safety.
• What does sim-to-real uncertainty do operationally? It shortens the trusted horizon, favors information gathering or conservative actions, triggers fallback, and selects real trajectories that should be added to training.