Object-centric spatial memory

Don’t start from a fashionable representation — start from the questions an agent must answer: what exists, where it is, which new observation belongs to it, and what stays true when it can’t be seen.

1 · Start from the question, not the representation

Your robot glances at a blue mug on the counter, a red bowl beside it, a closed cupboard. It turns toward the sink; the mug leaves the frame. A person walks between the robot and the counter. Five seconds later it must answer: “where is the mug, is the path to it clear, and what happens if I reach from the left?” The current frame has no mug in it at all — yet the answer is obvious to you, because you kept a thing in mind, not a picture. That thing-you-keep-in-mind is what this lesson builds.

First principles here means starting from the computation. A representation earns its keep by making frequent queries cheap and reliable, and this robot needs four kinds:

• Existence and identity: one mug or two, and is the reappearing blue patch the same mug?
• Metric space: where is it relative to the robot, the counter edge, and free space?
• Interaction: what holds it up, what blocks it, what would move if pushed?
• Persistence under missing evidence: what stays in memory while a surface is out of view?

None of those is answered well by an embedding of the latest frame. So a useful belief separates entities from the current image:

Each entity eⁱ carries a persistent identity ιⁱ, pose and velocity pⁱ,vⁱ, appearance/semantic features fⁱ, and uncertainty uⁱ; M is spatial memory (occupancy, free space), R is relations, g is global context. And it’s a belief, not an inventory — every part can be uncertain, and even the count N_t is a guess.

2 · Follow one mug to see why persistence is unavoidable

Trace the blue mug through a minimal filter. On first sight, the model opens a track with an estimated world position and covariance. While it’s visible, vision corrects that estimate. When the camera turns away, the model does not delete it — it predicts the entity forward (say, nearly static) while growing uncertainty, because someone could have moved it. When the mug reappears, the model compares the new detection against surviving hypotheses and either updates the old track or starts a new one:

The message mⁱ summarizes relevant interactions — a hand touching the mug should bend its predicted motion, a distant spoon should not — and the update U only fires when an observation j is assigned to entity i. So object permanence isn’t a slogan; concretely, it is the choice to keep predicting an entity’s belief when its observation likelihood is absent, widening uncertainty instead of faking certainty.

3 · Let query cost pick the representation

No state format wins everywhere, and “more explicit” isn’t automatically better — it can make updates brittle or slow. The honest way to choose is to derive each representation from the first query that becomes painful:

1. The entity question. “How many things, which is the mug, how will that one move?” On a dense image-like field you’d segment and search every time. That pain motivates a named, indexable unit — object slots or tokens make counting, attribute lookup, and per-entity prediction roughly O(N). The price: the model must decide where one entity ends and another begins, so merge, split, and identity-swap errors become the things to watch — and slots stay awkward for pixel-perfect rendering and amorphous background.
2. Add time. A fresh object set per frame answers “what’s here now” but not “where was this mug going, and how long unseen?” Recomputing history is slow and re-labels identity after every occlusion — which motivates tracking memory: keep trajectory, velocity, time-since-seen, and a track-ID lookup. Now temporal queries are cheap, but you inherit the detector’s errors, and unknown objects plus full free-space still need something else.
3. Ask an interaction question. “What supports, contains, or touches the mug?” Testing every geometric relation from scratch is wasteful and ambiguous under occlusion — which motivates a scene graph, turning the query into a local traversal around the mug’s node. The price is an explicit relation vocabulary and edge updates, with stale or impossible edges (and a too-rigid ontology) as the risks.
4. Ask the control loop’s repeated safety question. A planner checks thousands of points for free space and collision; an entity search per point is the wrong shape. A bird’s-eye-view or occupancy grid turns those into local array reads — fast, but position is quantized: thin geometry vanishes, vertical structure collapses, moving objects smear, and identity isn’t native. Cell size, cleared trails, and collision misses are the diagnostics.
5. Ask for an observation. Reconstruct appearance, or predict what a sensor would see somewhere. A short attribute list can’t reproduce texture, lighting, and everything unlabeled — which motivates a dense feature map or neural field, making local appearance native. But retrieving “the mug” again needs search, and identity can stay implicit even while renders look gorgeous.

The real answer is almost always hybrid: a metric map for navigation, object tracks for manipulable things, a graph for sparse interactions, and dense residual features for whatever the ontology misses. Duplication is fine if synchronization is explicit — when the mug moves, both its entity pose and its occupied cells must update. Judge a representation by total system cost (encoding, update, queries, inconsistency), not by latent size.

4 · Object discovery produces hypotheses, not ground truth

Where do entities come from? Supervised detections and masks, an unsupervised slot model whose vectors compete to explain regions, or a query-based transformer decoding a variable set of object hypotheses plus “no object.” Every route is an inference choice. Slot attention shows the mechanism: image features vote for slot queries, normalized attention makes slots compete, an iterative step aggregates each slot’s evidence, and a decoder renders a component and mask per slot to be composed. Reconstruction pressure encourages coverage; competition encourages separation. But reconstruction alone never says the mug should occupy the same slot tomorrow — a model can permute slots every frame and reproduce all pixels equally well.

5 · Binding is constrained inference

Two blue mugs cross paths. Next frame: two detections, two predicted tracks. Which measurement updates which track? Binding minimizes a cost that blends predicted position, appearance, shape, class, and motion consistency:

A one-to-one matcher minimizes total cost, plus “new entity” and “missed observation” options. Hard matching gives crisp identities but can make one irreversible mistake; soft attention stays differentiable but may blend two mugs into an average; a stronger belief keeps multiple association hypotheses alive until evidence decides — at more memory and compute. And the key insight: identity is not appearance. Lighting shifts, objects rotate, and identical manufactured mugs really are indistinguishable from one view. Identity is the continuity of a physical hypothesis through space, time, and interaction — if mug A is picked up while mug B stays on the counter, contact is stronger identity evidence than color. That’s why dynamics and relations should participate in matching, not be applied after it. Slot swapping — two indices trading physical referents — barely dents per-frame reconstruction, yet the dynamics head now hands A’s velocity and affordances to B. Diagnose it with identity survival through crossings and long occlusions, not adjacent-frame accuracy.

6 · Absence is evidence only when you expected to see it

A missing detection has several causes: the mug left the field of view, something occluded it, the detector failed, or it was removed. Treat all absence as nonexistence and you destroy permanence; treat all absence as harmless and you breed immortal ghosts. So the model predicts visibility from geometry. Let V_tⁱ mean entity i should be visible from the current pose, and split the update:

• Not in view: preserve it, evolve from dynamics; absence carries almost no information.
• Predicted occluded: preserve it, grow uncertainty, keep the occluder relation.
• Predicted visible but not detected: lower existence probability — this absence is informative.
• Reobserved: bind, correct, and check whether the uncertainty was calibrated.

In plain words: not seeing the mug matters only if the model believed it had a clear chance to see it. Track deletion should key on this visibility-aware existence belief, not a fixed count of missed frames — and for safety, a planner may keep a low-probability obstacle around longer than a low-probability decoration, because the cost of being wrong differs.

7 · Relations turn global dynamics into local computation

The mug’s next state depends mostly on nearby supports and contacts, not the whole scene. So build a time-varying graph — entities as nodes, candidate interactions as edges — and take a permutation-invariant message step:

The same rule works for any object count, so pushing the bowl instead of the mug reuses the same contact mechanism, and sparse neighborhoods cut all-pairs O(N²) down to roughly O(|E|). But edges are themselves beliefs — contact starts and stops, support transfers from counter to gripper, containment hides — so predict relation probabilities and let geometry prune impossible edges rather than assuming a fixed graph. Relations also protect binding: a hand and mug moving together after contact is identity evidence, while an impossible teleport or sudden support change more likely signals an identity swap than exotic physics.

8 · Training must reward persistence, not just pixels

A reconstruction loss alone permits temporally inconsistent decompositions, so the training signals must target the behaviors the memory exists for:

• Observation reconstruction: decoded entities and background explain visible RGB, depth, masks, or flow.
• Temporal consistency: matched entities predict future pose and appearance; contrastive terms pull one track together and push others apart.
• Set supervision: permutation-invariant matching compares predicted and labeled entity sets without reading meaning into slot index.
• Occlusion curriculum: mask objects or use long natural occlusions, then score identity and calibrated existence on return.
• Dynamics and relation loss: predict action effects, contact, support changes, and future occupancy.
• Task loss: train or evaluate planning queries — collision, retrieval, manipulation success — so a beautiful decomposition isn’t the only goal.

A common joint objective is L = L_obs + αL_set + βL_dyn + γL_id + δL_task, and the weights matter: too much per-pixel pressure spends slots on texture; too much identity regularization resists legitimate split/merge events like opening a cupboard. Staged training helps — learn perception, add temporal matching, then fine-tune dynamics and task heads while continually testing the whole loop.

9 · How many objects, and for how long — cardinality is part of the state

The entity set looked fixed, but real worlds have births, exits, assembly, separation, and shifting relevance. A person carries in a plate; the robot pours several ice cubes; a cupboard opens to reveal objects that existed before they were seen. The system must tell new to the world from new to the sensor — and usually it can’t know at once, so it should hold a tentative birth rather than force a binary. That gives a lifecycle: an unmatched detection starts as a low-confidence candidate; repeated compatible evidence promotes it to an active track; missed evidence sends it dormant rather than deleted; later it can reactivate, archive, merge with a duplicate, or terminate when visible absence and exit geometry make disappearance likely — with thresholds tuned to detection reliability and task risk. Existence itself is a tracked probability:

Variable cardinality strains architectures. Fixed slots batch efficiently but need an explicit presence variable and overflow in crowds; a large fixed budget wastes compute and breeds duplicates; dynamic query sets fit cardinality naturally but complicate batching. Many systems keep a moderate active set, summarize distant objects into a map, and reactivate tracks when the camera returns. Split and merge need semantic care: separating regions may be two revealed objects, two parts of one articulated body, or one object that broke. Rather than overload slot count with meaning, represent hierarchical entities and relations — the cupboard is an entity, its door and body persistent parts joined by a hinge; the mug and saucer separate entities linked by support — so navigation can treat a stack as one obstacle while manipulation keeps two identities. Background needs memory too: a slow-changing spatial layer, active dynamic entities, and a residual appearance field, with promotion from residual to entity when an unexplained region starts moving or becomes task-relevant.

10 · Evaluate the exact invariance memory promises

A single detection metric can’t measure a persistent world belief, so build evaluation around controlled transformations. For viewpoint invariance, orbit a static kitchen and verify world-frame entities stay put while visibility changes. For identity, cross visually identical objects, fully occlude one, and score which history each reappearing object inherits. For existence calibration, group predictions by confidence and check whether objects actually survive at the advertised rate. For interaction, intervene on one member of a relation and confirm only causally connected entities respond.

Evaluate queries at their native operating point: collision false-negatives for the BEV at the robot’s safety margin, retrieval latency as object count grows, position error versus occlusion duration, planning success from the stored state. If a representation quantizes space to save memory, plot downstream failures against cell size. Counterfactual corruption is especially telling — swap appearance while keeping the trajectory (does identity follow continuity or color?), perturb camera pose while keeping pixels (does the map reveal an inconsistent transform?), delete the observation of a visible mug (does existence fall more than under an occluder?). And match set metrics carefully: detection precision may count a duplicate track as one extra false positive while the planner sees two obstacles, so report duplicate rate, fragmentation, merge rate, identity switches, and time-to-reacquire, stratified by occlusion length, similarity, speed, and crowding. An aggregate score otherwise rewards easy visible objects and hides the persistence problem the representation was meant to solve. Finally, test compositional generalization — train interactions with two or three objects, then evaluate more objects and novel arrangements; a state whose “names” are truly compositional should degrade gradually, not collapse because the mug is green.

11 · Diagnose the mechanism, not the final score

A final score says something downstream failed; it doesn’t say which update rule to fix. Diagnose linearly: hold the scene fixed, vary one source of evidence, find the first internal state that goes wrong, and repair that before retraining everything.

1. The mug vanishes whenever the robot turns. The mug didn’t disappear; the camera lost access. A frame-only state, or a deletion rule that ignores predicted visibility, confuses “not measured” with “doesn’t exist.” Hold out longer out-of-view intervals and inspect the track before return; the fix is a world-frame track that survives while uncertainty grows, and it passes only if reobservation corrects that same hypothesis rather than spawning a new mug.
2. Two blue mugs swap trajectories. The matcher trusted appearance even when paths and contacts disagreed. Recreate a crossing while controlling color, motion, and hand contact; if identity follows color, binding is appearance-dominated. Add motion and relation evidence (or keep multiple hypotheses until the crossing resolves) and measure identity survival, not adjacent-frame overlap.
3. A ghost mug blocks the planner forever. Persistence cured premature deletion but made existence immortal. Put the predicted mug in a clear, visible spot and repeatedly fail to detect it — that absence should drop survival far more than absence behind a wall. Make expected visibility negative evidence and calibrate the resulting probability against actual survival.
4. The occupancy map leaves a smeared trail. Every observed occupied cell got fused into a “static” layer, so old mug cells stayed occupied after it moved. Replay one controlled translation and inspect map layers: separate static structure from moving-entity occupancy, update the dynamic layer from the track, and explicitly clear vacated cells.
5. Rendering looks great, grasping fails. The objective rewarded pixels, so the state learned texture without the metric pose, support, and contact precision the controller needs. Hold appearance constant and probe those variables directly; add explicit geometry or a task loss at the failing scale and verify grasp success, not image quality.
6. Memory grows without bound. The model creates and keeps tracks but has no lifecycle policy for duplicates or low-value entities. Drive a long sequence and plot active state by age, confidence, and task relevance; merge duplicates, archive low-value tracks, keep uncertainty summaries by decision risk. The test is graceful bounded growth without forgetting safety-critical entities.

12 · Product and system tradeoffs

Structured memory brings operational duties. Variable entity counts complicate batching; association adds latency; a high-resolution BEV makes collision checks cheap but eats memory over large areas; raw keyframes support re-identification but raise bandwidth and privacy costs; learned slots cut labeling but are harder to debug than detector-backed tracks. So use a tiered memory: fast local occupancy and active tracks at control rate, dense appearance or a global map updated more slowly, uncertain history archived for recovery rather than sitting in every planning step. Attach a timestamp, confidence, provenance, and coordinate frame to every fact so that when sensors disagree the system can retract an update instead of silently averaging incompatible states. Tune lifecycle thresholds to decision cost — a home robot conservatively keeps a possible child or pet as an obstacle but happily forgets a spoon’s color; “what to remember” is a product policy, not just a representation choice.

Make the interface force uncertainty into the open. Instead of get_pose(id), return a pose distribution, timestamp, visibility state, and source, so a planner can enlarge a collision envelope for a stale track, request a fresh view before a delicate grasp, or reject an entity inferred from an untrusted sensor — preventing downstream code from turning a probabilistic memory into accidental certainty. Privacy shapes design too: persistent appearance embeddings can re-identify people, so if the task only needs occupancy and coarse motion, drop identity-rich features after association or keep them on-device briefly. Entity memory actually makes selective deletion easier than an opaque history embedding — remove one person’s appearance while keeping anonymous free-space evidence — but duplicated state means deletion must propagate to maps, tracks, retrieval stores, and logs. And measure behavior under load: when object count spikes, a deterministic policy should prioritize safety-critical classes and nearby interactions, summarize distant ones, and expose overload to the planner. An elegant all-pairs model that misses control deadlines is not useful state.

13 · Bridge: objects need a stable place to live

Object-centric belief fixes one problem — the mug persists as a thing instead of being rediscovered each frame. It exposes the next: coordinates. Store its position in camera pixels and turning the camera makes every stationary entity appear to move. Lesson 11 separates sensor pose from world state and asks when explicit 3D/4D geometry makes prediction and planning queries cheaper.

Interview prompts

• What is object permanence computationally? It is predicting an entity belief when its measurement is absent, preserving identity while increasing uncertainty, then correcting that same hypothesis when evidence returns.
• Why is object discovery not enough for tracking? A per-frame decomposition is permutation-invariant. Tracking additionally binds current observations to persistent physical hypotheses using motion, appearance, geometry, and relations.
• When should absence delete an object? Absence is strong evidence only when the object was predicted to be visible and the detector had a good chance to see it. Out-of-view or geometrically occluded absence should mainly widen uncertainty.
• How would you choose slots versus BEV? By query cost: slots make count, attributes, and per-object dynamics cheap; BEV makes free-space and collision queries cheap. Many systems need both.
• What is slot swapping, and why can reconstruction miss it? Two latent indices exchange physical identities. The same components can still reconstruct both objects, but their histories, velocities, and affordances become attached to the wrong thing.
• Why use graph dynamics? Most interactions are local. Shared message functions provide compositionality and permutation symmetry, while sparse edges avoid all-pairs computation.
• When is an object decomposition a harmful bias? When dynamics depend on amorphous fields, material continua, crowds, or task-defined regions without stable boundaries. Hybrid entity-plus-dense states reduce the mismatch.
• How would you evaluate memory beyond detection accuracy? Test identity through crossings and long occlusions, existence calibration, relation transitions, map consistency, query latency, and whether plans based on the memory succeed.