Partial observability and belief state

A single sensor reading almost never pins down one world. So instead of guessing a state, the agent should carry a whole distribution over what the state could be — push it forward when it acts, sharpen it when it sees, and keep the ambiguity that changes what it should do.

1 · One frame is not a state

Watch a ball roll behind a wall. Right now, the camera shows nothing but wall — no ball at all. Yet you are not confused: you know a ball is back there, roughly where it will reappear and when. The frame threw that information away; your memory kept it. That gap is the whole subject of this lesson.

Let’s say precisely what the frame is missing. A state is Markov when knowing it makes the past irrelevant — once you have s_t and the action a_t, older history tells you nothing more about what comes next:

The current image almost never has this property. When the ball is behind the wall, the frame has no ball pixels, yet its earlier position and speed still decide where it emerges. Two different histories can end in the exact same “empty wall” picture and then diverge completely. Collapsing both into the same frame-only representation is a specific, nameable mistake — state aliasing: genuinely different states handed the same internal label.

Aliasing sneaks in through several doors:

• Occlusion: things keep existing when another surface hides them.
• Projection: many 3D scenes flatten to the same 2D image. The 3D Vision track studies these geometric ambiguities.
• Hidden derivatives: one position never reveals velocity or acceleration.
• Unobserved properties: mass, friction, charge, and intent often show themselves only through interaction.
• Sensor limits: noise, finite resolution, exposure, missing modalities, and timing all erase information.

The formal name for this world is a partially observable Markov decision process (POMDP). The physics can be perfectly Markov while the agent’s view of it is not:

Read that inequality as a permission slip and a warning at once: older evidence can change what the next observation means, so the image is not useless — it just has to be combined with what you remember and a model of how the world moves.

2 · Hold a belief, not a guess

If you can’t read the true state, what should you carry instead? Not a single best guess — that throws away the alternatives. Carry a ranked list of suspects with probabilities. Formally, a posterior over what the state could be, given everything you’ve seen and done:

Read it right to left. The conditioning bar lists your evidence: every observation up to now and every action you already took. The left side, b_t(s), hands each candidate state a probability. This is called the belief, and it is not fuzziness about a neural output — it is the agent’s knowledge of the world.

Here is why the belief is the right object. Two histories that produce the same belief lead to the same future under the same actions (if the model is right). So the belief quietly converts a hard partially-observed problem into an ordinary fully-observed one — just played in belief space. The controller’s “state” is now a distribution.

It also forces apart two kinds of not-knowing, because they need different cures. Aleatoric uncertainty is genuine ambiguity that survives even a perfect model: the ball really could be behind either door, and more of the same camera won’t settle it — only a new kind of look will. Epistemic uncertainty is ignorance about the model itself: maybe this warehouse corner was barely in the training data. More diverse data or an ensemble shrinks that; more glimpses of the same scene does not. Keep them separate and you’ll reach for the right remedy.

3 · Every update is two moves: predict, then correct

Updating a belief is always the same two-step dance, and always in the same order: first move your belief because you acted, then reshape it because you saw. Start from last step’s belief b_t−1.

Move A — predict through the action

The bar means “before seeing anything new.” In words: for each candidate new state s′, ask how much old belief could have flowed into it — take each old state’s probability, multiply by the chance the action carried it to s′, and add those up. (For continuous states the sum is an integral.) Prediction usually spreads the belief out, because one state can lead to several.

Move B — correct with the observation

Here O(o_t | s′) is the likelihood — how well candidate s′ explains what you just saw. Multiplying by it lifts states that fit the evidence and pushes down states that don’t; η just rescales everything to sum to one. Correction usually sharpens the belief — unless the observation is noisy or ambiguous, in which case it barely moves.

Order is not a technicality. The action happened before the measurement, so belief must move by the action first. Correct “yesterday’s state” with “today’s observation” and only then transition, and you’ve scrambled your time indices — a small slip that becomes a real bug the moment sensors lag or run at different rates.

4 · Do it by hand: which aisle has the forklift?

Nothing makes the two moves click like running them once with real numbers. A robot faces two blind aisles, left (L) and right (R); a forklift sits in one. Routing history gives a prior P(L)=0.60, P(R)=0.40. A noisy microphone names the right aisle 80% of the time: P(report L | L)=0.8, P(report L | R)=0.2.

First report: “left”

Correction is prior × likelihood. Left: 0.60 × 0.80 = 0.48. Right: 0.40 × 0.20 = 0.08. They sum to 0.56, so normalize:

One report lifts left from 60% to about 86% — strong, but not certain, because the sensor sometimes lies.

Second report: “left” again

Yesterday’s posterior becomes today’s prior. Left weight: 0.857 × 0.8 = 0.686; right weight: 0.143 × 0.2 = 0.0286. Normalize:

Independent evidence stacks. There’s an even cleaner way to see it, in odds: prior odds are 0.6 / 0.4 = 1.5, each “left” report multiplies by its likelihood ratio 0.8 / 0.2 = 4, so two reports give 1.5 × 4 × 4 = 24 — probability 24 / 25 = 0.96. Same answer, less arithmetic.

Then the camera blinks: the forklift can move

Now one step passes with no look, and the forklift stays put with probability 0.9, switches with 0.1. That’s a predict step:

Notice belief decayed from 0.96 to 0.868 with no new evidence — exactly right, because the world might have moved. This isn’t the model getting confused; it’s honesty about a stochastic transition.

A contradiction arrives: “right”

The next report says right. For left that report has likelihood 0.2; for right, 0.8. Weights: 0.868 × 0.2 = 0.1736 and 0.132 × 0.8 = 0.1056. Normalize:

Belief is still slightly left — which is exactly the judgment you want. One noisy contradiction shouldn’t erase two consistent reports, but it should knock confidence down hard. A rule that remembers only the latest report would flip to right; a rule that locked in after two reports would stay at 96%. Bayesian filtering is what sits sensibly between those two failures.

What the belief buys you

Turn it into a decision. If entering an occupied aisle costs 100 and a careful visual check costs 8, then taking the right aisle now has expected cost P(R) × 100 = 0.378 × 100 = 37.8; taking left costs 62.2; waiting costs 8 — so gathering information is the rational move. Change the wait cost to 50 and taking right wins instead. The belief never decides on its own; it hands the objective the probabilities it needs.

The widget tracks entropy — one number for how spread-out the belief is. For two states it peaks at 50/50 and falls toward zero as one hypothesis wins. Notice it answers a different question than “which door leads”: the top probability says who’s winning; entropy says how sure you are overall. A good decision often needs both — the leading guess and how much to trust it against the cost of being wrong.

5 · In the real world, the belief is only ever approximate

The two-aisle belief was two numbers. A real visual world hides continuous poses, velocities, identities, shapes, intentions, and maps — you could never list every possible state, let alone score it. So every real system approximates the posterior. The useful question isn’t “neural or classical?” It’s: what shape of doubt must survive? Pick the cheapest tool that keeps that shape.

1. One smooth cloud. If the state moves roughly linearly and the noise is roughly Gaussian, a Kalman filter needs only a mean and a covariance — the center of the cloud and how it stretches. Under those assumptions it’s exact, cheap, and readable. The trap: keep using one cloud when the forklift could be in either aisle, and its mean lands between them — in the wall — erasing the two options the decision actually turns on.
2. A gently bending cloud. Extended and unscented filters still carry a mean and covariance but push them through nonlinear motion with a local linearization or a few probe points. Fine for mild curves. They break when a transition folds one blob into two sharp modes — a couple of local summaries can’t represent a shape they were never built for.
3. Truly separate possibilities. A particle filter is just a swarm of weighted guesses. Particles can follow nonlinear motion and sit in several modes at once, so “left aisle” and “right aisle” stay physically real instead of averaging into a wall. The cost is honest: rich worlds need many particles, and careless resampling can quietly delete a rare mode that later turns out to be the dangerous one.
4. Let the network keep the memory. An RNN, GRU, or LSTM folds observations and actions into a hidden vector, fast and able to hold evidence across long gaps. But a plain vector doesn’t automatically expose a distribution — its uncertainty may be implicit, unreadable by the controller, and poorly calibrated even when the average prediction is good.
5. Give learned memory a place to branch. A stochastic state-space model pairs deterministic memory (temporal context) with random latents (sampled alternatives), so “what the history says” and “which unresolved future happens” live in separate slots. Still, the posterior family limits which ambiguities fit, and a careless objective can collapse the modes until the random part is ignored.
6. Reach far back by content. Transformer memory keeps tokens in a context window or cache and attends straight to whatever past evidence matters now. That reach isn’t free: context length, compute, positional bookkeeping, and evicting stale tokens all become real constraints.

Read that list as a diagnosis, not a menu. One near-Gaussian mode → mean and covariance. Curved-but-single doubt → nonlinear Gaussian filtering. Separated hypotheses → samples, mixtures, or stochastic latents. A decisive clue buried deep in history → recurrence or retrieval. Reach for the fancier tool only when the simpler one destroys a distinction that would have changed a decision.

And a learned model needn’t reproduce Bayes’ rule symbol for symbol — it just has to carry the same information flow: before a new observation, advance memory by the last state and action; after it arrives, compare prediction to evidence and revise. In state-space models these are the prior and posterior pathways, and training pulls them together where the world is predictable while letting the posterior fix surprises.

6 · A single “best estimate” can be a world that can’t exist

Say the forklift is equally likely 4 m down the left aisle or 4 m down the right. Average those coordinates and you place it in the wall between them — a state with essentially zero probability. That is the core hazard of squashing a multi-peaked belief to its mean, and mean-squared error actively encourages it, because the conditional mean is exactly what minimizes squared loss.

So match the summary to the question:

• The mean is fine for near-Gaussian tracking with quadratic costs.
• The most-likely hypothesis gives one coherent world but silently drops the rest.
• Modes or particles keep mutually exclusive futures alive for risk-aware planning.
• Quantiles or tail probabilities speak directly to safety limits like “collision chance under 0.1%.”

7 · A long memory is not a good memory

Holding 10,000 tokens guarantees nothing — the frame that mattered might be crowded out by 9,999 that didn’t. A real belief needs selective memory: keep the pedestrian who ducked behind a bus, drop the sensor flicker that changes no future. And it needs selective revision: don’t cling to a stale pose once strong evidence contradicts it.

You can probe for both directly:

1. Occlusion sweep: hide an object longer and longer, and watch how state and outcome error grow.
2. Contradiction: confirm confidence falls before the estimate flips.
3. Action-conditioned update: check that the robot’s own motion shifts the predicted scene before the next frame arrives.
4. Distractors: inject flashy but irrelevant events and see whether the target is still remembered.
5. Information probe: try to decode hidden position, velocity, identity, and uncertainty straight from the frozen state.

If a hidden vector predicts well but a simple probe can’t read its uncertainty, the planner won’t be able to use that uncertainty either. Representation and interface have to be designed together.

8 · Some actions don’t change the world — they change what you can see

In a fully observed problem you judge an action only by how it moves the physical world. Under partial observation, an action can also move what becomes knowable: peek around an occluder, slow down for more frames, tap an object to feel its mass, light up a dark corner, ask the one question that splits two hypotheses.

The value of information isn’t “reduce uncertainty for its own sake.” It’s how much a possible observation improves your expected decision, minus what it costs to get. In the aisle example, waiting was worth at most the gap between the best immediate expected cost (37.8) and the expected cost after checking. If a perfect check always lets the robot pick the empty aisle for free, information is worth up to 37.8 — so an 8-unit wait pays off, a 50-unit wait doesn’t.

That opens a second loop stacked on the first: use the belief to choose an informative action, take in the evidence, update, then choose the task action. A model that hides its ambiguity inside visually varied samples — but can’t attach probabilities or expected consequences — makes this loop almost impossible to optimize.

9 · Two sensors help — unless they’re wrong the same way

Extra sensors cut ambiguity because they constrain different parts of the state: cameras give dense appearance but fail in glare; radar gives range and speed at another resolution; proprioception reports the agent’s own motion. Fuse them in a common time and coordinate frame, each weighted by how reliable it is.

The clean case assumes the sensors are independent given the state: p(o¹,o² | s)=p(o¹ | s)p(o² | s), so you just multiply their likelihoods during correction. But that assumption is often a lie. Two camera detectors trained on the same data fail together in fog; two “independent” location estimates may both already contain the same GPS fix. Treat correlated evidence as independent and you count the same clue twice — and become overconfident.

The numbers make it vivid. Start at 50/50 for a pedestrian crossing. A vision classifier and a “second” classifier each shout cross with likelihood ratio 4. If truly independent, posterior odds are 1 × 4 × 4 = 16 — probability 16/17 ≈ 94.1%. But if the second is just a copy fed the same image features, it adds almost nothing, and the honest answer stays near 4/5 = 80%. That gap between 94% and 80% is why modeling correlation is a safety issue, not statistical polish.

Timing bites too. If a camera frame is 100 ms stale while proprioception is current, fusing them as if simultaneous drops objects into the wrong ego frame. The fix is disciplined bookkeeping: timestamp every observation, transform it through estimated ego motion, update the belief at the measurement’s time, then roll forward again. Same story for delayed messages and rolling-shutter cameras. And a missing sensor should widen the belief, not be replaced by a confident default — train with modality dropout and validity masks so the filter degrades gracefully instead of lying.

10 · Planning happens in belief space

A physical action changes the distribution of future states; an observation changes the distribution of beliefs. So under partial observation a policy is really a function of the belief, a_t = π(b_t), not of the single most-likely state. Two beliefs with the same center but different spread may call for different speeds; two equally-spread but differently-shaped beliefs may call for different peeks.

Exact belief-space planning is expensive — every candidate action branches into possible states and possible observations. Real systems approximate: search over sampled hypotheses, use certainty-equivalent control with a safety margin, learn a policy from belief features, or alternate planning with targeted looking. The one non-negotiable interface: the controller must receive uncertainty in a form its objective can actually use.

11 · How belief filters fail (and the fix)

• Overconfident transition: belief doesn’t spread while unobserved. Remedy with stochastic dynamics, ensembles, and calibration over longer gaps.
• Overreactive correction: one noisy reading wipes out strong history. Train with sensor corruption and expose likelihood/reliability where you can.
• Underreactive correction: recurrent memory shrugs off a surprise and goes stale. Test abrupt changes; add update gates or explicit posterior correction.
• Mode collapse: several live states become one mean or one arbitrary sample. Score mode coverage and decision tails, not just the best sample.
• Filter divergence: a wrong model turns confident, so real evidence looks impossible and gets rejected. Watch the innovation — predicted minus observed — and keep a recovery path.
• Confusing the two uncertainties: sampling more futures from one confident-but-wrong model does not reveal model ignorance. That needs ensembles, out-of-distribution signals, and coverage tests.
• Ignoring timing: applying stale observations to the current state conjures ghost motion. Track timestamps; update the past state, then replay.

12 · How to reason about it out loud

For a partially-observed design question, answer in a fixed order and you’ll never flail. First, name the hidden variables and an aliasing example. Second, say what history actually arrives, including action and sensor timing. Third, choose a belief representation that can hold the posterior shape you expect. Fourth, spell out the predict and correct paths. Fifth, say how uncertainty reaches the planner. Last, propose the tests — calibration, occlusion, contradiction, and a downstream decision.

Concretely: a robot with cameras and proprioception might keep deterministic recurrent memory for long-range context, stochastic object states for ambiguous positions and intentions, and an explicit ego-pose covariance from geometric localization. Predict uses commanded and measured motion; correct fuses new image features; an occupancy head returns probabilities at future horizons; the planner maximizes expected progress under a collision-probability limit, executes one short segment, looks again, and replans. It hangs together because every piece has a job in the information flow.

13 · Takeaway

But the exact posterior over a whole visual world is astronomically large — you can’t store or update it directly. So Lesson 03 asks the next forced question: how do you compress history into a compact learned state that still keeps the distinctions and uncertainty your questions depend on?

Interview prompts

• Why is video usually non-Markov in pixel space? Current pixels omit velocity, occluded entities, 3D causes, and hidden properties, so older evidence still changes the next-frame distribution.
• What are the two Bayesian filtering steps? Predict the previous belief through the action-conditioned transition, then correct the resulting prior using the new observation likelihood.
• What does the normalization constant do? It rescales likelihood-weighted prior masses so the posterior sums or integrates to one.
• When is a particle belief preferable to a Gaussian? When the posterior is nonlinear, skewed, or has separated modes such as mutually exclusive locations.
• Why can the posterior mean be invalid? For separated hypotheses it may lie in a low-probability or physically impossible state between the modes.
• How do you test uncertainty calibration? Group predictions by reported probability and compare confidence with empirical frequency, sliced by horizon and difficult conditions.
• What is an information-gathering action? An action chosen partly because its possible observations improve a later decision enough to justify its cost.
• How are aleatoric and epistemic uncertainty different? Aleatoric uncertainty is unresolved outcome variability under the model; epistemic uncertainty reflects ignorance about the model and can shrink with better coverage.