Learned depth — monocular, self-supervised, and MVS

Everything so far needed a crutch: a depth sensor (lesson 03), multiple posed views (lesson 05 and CV 05–07), or 3D labels to supervise a network (lesson 11). None of those is an ordinary photograph. So the question this lesson forces: can we recover 3D from a single, uncalibrated, unposed image — with learning? Monocular depth is ill-posed — it is exactly the scale ambiguity of CV 04/05 and lesson 01's inverse-rendering problem, where a small near object and a large far one are indistinguishable. There is nothing to invert from one view. A network escapes only by supplying learned priors. We cover three regimes: supervised monocular depth (and why "metric" is different from "relative"), the elegant self-supervised loop that learns depth from video with no labels at all, and learned multi-view stereo when you do have several views and want dense geometry.

1 · Why monocular depth needs priors

Start from the fact CV 04 proved and lesson 01 built the whole track on: projection divides by depth. A pixel x = f·X/Z fixes a ray, not a point — scale the world point along its ray to (tX, tY, tZ) and the t cancels, so every depth along the ray lands on the same pixel. For a single image this means: every pixel's depth is a free variable. A depth map assigns one number per pixel; nothing in the image constrains it. Formally, given one image I there are infinitely many depth maps D such that back-projecting (lesson 03: p_cam = D·K^-1[u,v,1]^ᵀ) and re-projecting reproduces I. The extreme case is global scale: a photograph of a dollhouse and a photograph of a real house can be pixel-identical (this is why film miniatures work). Depth-from-one-view is not under-determined by a little — it is under-determined by an entire function.

So how can a network output a depth map at all? Not by inverting geometry — there is nothing to invert from one view, no second ray to intersect (contrast CV 05's triangulation, which needs two). It escapes the ambiguity the only way possible: by learning priors — regularities of how real scenes look, absorbed from training data. The cues it learns are exactly the ones human monocular vision uses:

• Perspective convergence. Parallel lines (rails, corridors, ceilings) meet at a vanishing point; the rate of convergence encodes surface orientation and depth.
• Texture gradients. A textured surface (gravel, grass, tiling) has finer, denser texture as it recedes — a monotone depth cue.
• Known object sizes. A person is ~1.8 m, a car ~1.5 m tall. Recognize the object, read its pixel height, and Z = f·(real size)/(pixel size) pins depth — if you know both the size and f. This is the object-size prior CV 04 flagged for monocular scale.
• Occlusion & defocus. What overlaps what gives an ordering (an occluder is nearer); the lens's depth-of-field blur is a weak metric cue.

The consequence to internalize: monocular depth is pattern-matching learned scene statistics, not solving a geometry problem. That is why it generalizes imperfectly (a scene unlike anything in training gets guessed badly) and can be actively fooled by unusual scale cues — a forced-perspective photo, a giant prop, an Ames room — where the learned priors point confidently the wrong way. The network is answering "what depth map is most likely, given scenes I've seen?", which is a probabilistic answer to a question that has no unique deterministic one.

2 · Supervised monocular depth, and metric vs relative

The supervised recipe is the obvious one: collect images each paired with a ground-truth depth map (from LiDAR, RGB-D, or a stereo rig), and train a network — an encoder-decoder or, in the modern form, a dense-prediction transformer (DPT) — to regress D = g_θ(I), minimizing a per-pixel depth loss. This works within a fixed domain. The difficulty is that ground-truth depth is scene- and sensor-specific: indoor RGB-D is in meters at one scale, driving LiDAR at another, and each dataset has its own camera. Naïvely regressing absolute meters ties the network to one sensor and one scene scale, and it fails to transfer.

The scale-invariant loss. The fix that unlocked cross-dataset training (MiDaS, then DPT) is to stop asking for absolute depth and instead compare prediction to ground truth up to an unknown global scale and shift. Concretely, before computing the error you align the prediction D to the target D^* by the best affine fit — solve for the scalar s and offset t that minimize ‖(sD + t) − D^*‖ (a 2-parameter least squares, in closed form) — and penalize only the residual:

Because the global scale and shift are factored out before scoring, the network is free to output depth in its own arbitrary units, and datasets at wildly different scales can be mixed in one training run. What the network learns is therefore relative depth — the ordering and relative structure of the scene (what is nearer than what, and by roughly how much in affine-invariant terms), not meters. This is often called ordinal or affine-invariant depth. Depth Anything pushes the same idea to scale by pseudo-labeling: a teacher model trained on the labeled data annotates millions of unlabeled internet images, and a student trains on that vastly larger, more diverse pseudo-labeled set — a semi-supervised trick that dramatically improves generalization (it is the depth-flavored cousin of the self-training in CV 14).

Getting back to meters needs a metric anchor; focal length matters, but is not universally an explicit input. Relative depth cannot become metric by posturing more confidently: its objective deliberately factored scale away. A metric model instead learns from depth measured in meters and from dataset priors about object sizes and scene layout. Camera intrinsics determine the ray associated with each pixel — recall x = f·X/Z — so a change in focal length can mimic a change in size or distance. But knowing f alone still does not reveal an object's unknown physical size, and a network trained on a fixed camera can absorb that camera model implicitly. Across variable cameras, robust systems handle intrinsics in several ways: pass K explicitly, canonicalize images/depth to a reference focal length, estimate intrinsics jointly, or learn the limited camera distribution present in training. A wrong camera assumption can create a systematic scale error, but “metric output” does not logically imply “focal length must be an input tensor.”

Which you want is set by the task, not by benchmark numbers. Relative depth suffices — and is more robust — when you only need ordering or relative structure: computational-photography effects (portrait-mode background blur, relighting), depth-aware compositing, or seeding a novel-view generator. Metric depth is mandatory the moment a real length enters the loop: robot grasping and obstacle distances, AR object placement that must sit correctly on a real table, or any measurement. Reaching for a relative-depth model where you need meters is a common and quiet mistake — it will look plausible and be unusably off in scale.

3 · Self-supervised depth from video — geometry as the loss

This is the elegant one, and the payoff of the whole track's framing. Ground-truth depth is expensive and sensor-bound; what if the only supervision were ordinary video — a moving camera, no labels, no poses? The Monodepth2 / SfM-Learner loop does exactly this, and the trick is to make geometry itself the loss. Two networks, trained jointly:

• DepthNet D = g_θ(I_t) — predicts a dense depth map of the target frame I_t.
• PoseNet T = h_φ(I_t, I_s) ∈ SE(3) — predicts the relative camera pose (lesson 02) from the target frame to a nearby source frame I_s (e.g. the previous/next frame).

The warp. With a depth for every target pixel and the relative pose, we can synthesize the target frame out of a source frame — and this synthesis chains together every operation from earlier lessons. For each target pixel p:

1. Back-project through the intrinsics and its predicted depth into a 3D point in the target camera's frame (lesson 03): P = D(p)·K^-1\,p.
2. Transform by the predicted relative pose (lesson 02) into the source camera's frame: P' = T\,P = R\,P + t.
3. Reproject that 3D point into the source image (the projection geometry of CV 04/05): p_s ∼ K\,P'.
4. Sample the source image's color at p_s (bilinearly, so the whole warp is differentiable) and paint it at p in the reconstructed target Î_t.

Do this for every pixel and you have re-rendered the target view from the source view's pixels, using only the predicted depth and pose. The single warp that maps a target pixel to its source location is:

The loss. If — and only if — the predicted depth and pose are correct, the reconstructed target Î_t matches the real target I_t. So the supervision is the photometric error between them (in practice a blend of an L1 term and structural similarity), plus an edge-aware smoothness regularizer that lets depth vary sharply only where image gradients are strong (object boundaries) and stay smooth on flat surfaces:

Both networks are trained by backprop through this differentiable warp — no depth labels, no pose labels, ever. The correct depth and pose are simply the ones that best explain how pixels actually move between consecutive frames. Read this against lesson 01's spine: the representation is (depth map + relative pose), the renderer is the warp-and-sample above, the loss is photometric, and the update is gradient descent through it. Self-supervised depth is the inverse-rendering loop with a depth+pose representation — the same loop as NeRF (lesson 09), with a far cheaper renderer.

Two wrinkles break the naïve version, and the fixes are instructive because they're pure geometry, not hacks:

• Occlusion — the per-pixel minimum reprojection error. A surface visible in the target may be hidden in a given source frame (something moved in front of it, or it left the frame). Warping into that source samples the wrong color and injects a large spurious error. The fix (Monodepth2): use multiple source frames (e.g. previous and next) and, at each target pixel, take the minimum photometric error over sources rather than the average. A pixel occluded in one frame is typically visible in another, and the min picks that good view — averaging would let the occluded frame poison the gradient.
• Moving objects — auto-masking. The whole loop assumes a static scene moved past by the camera. An object moving with the camera (a car ahead at the same speed) appears not to move between frames; the only depth that explains "no apparent motion" is infinite, so the network paints it at infinite distance — the notorious "holes of infinite depth." The fix is auto-masking: drop any pixel whose photometric error against the unwarped source (identity warp) is already lower than against the warped source. If moving nothing explains a pixel better than the predicted motion does, that pixel is not obeying the static-scene assumption, so exclude it from the loss.

4 · Learned multi-view stereo (MVS)

Monocular depth trades geometric evidence for needing only one image. When you have several views, correspondences and parallax constrain depth directly. Keep the history straight: feature-based SLAM / SfM commonly builds a sparse map for localization, but classical multi-view stereo was already dense long before deep learning — plane sweeps, photoconsistency, graph-cut/variational methods, and PatchMatch MVS all produced dense surfaces from calibrated views. Learned MVS does not invent density; it replaces brittle hand-designed matching and regularization with learned features and priors. The cleanest bridge is still the plane sweep.

Plane sweep. Fix a reference view. Hypothesize a set of fronto-parallel depth planes at candidate depths {Z₁, …, Z_K} (usually sampled uniformly in inverse depth, since depth resolution degrades like Z² — the lesson 03 / CV 05 reason). For each hypothesized depth Z, warp every neighboring view onto that plane and back into the reference frame. Warping an image via a plane is a homography H(Z) (CV 05: a plane induces a homography between two views), parameterized by the known relative poses and intrinsics. The signal is this: at the true depth, all the warped neighbor views agree with the reference (they are all imaging the same real surface point), so the photometric variance across views is low / the matching score is high; at a wrong depth they disagree.

Cost volume. Stack the per-plane matching costs into a 3D cost volume of shape H × W × K — for each reference pixel and each depth hypothesis, a scalar cost (built in practice from the variance of deep features across the warped views, not raw pixels). This volume is noisy and has ambiguities (textureless regions, repeats). So regularize it with a 3D CNN that smooths and sharpens along the spatial and depth axes, enforcing that neighboring pixels have consistent, spatially coherent depth. Finally, collapse the depth axis to a single depth per pixel with a soft-argmin — a softmax over the (negated) cost along depth, then the cost-weighted expected depth:

The soft-argmin is differentiable (unlike a hard argmax), giving sub-plane depth precision and letting the whole thing — feature extractor, cost volume, 3D CNN — train end-to-end against ground-truth depth. This is the MVSNet family. Run it per reference view and fuse the resulting depth maps into a single point cloud or mesh by back-projecting each (lesson 03) and merging with a cross-view consistency check that rejects points not confirmed by enough neighbors — landing back on the point-cloud/mesh representations of lessons 03 and 05. A closely related task is depth completion: given a sparse LiDAR sweep and an aligned RGB image, a network fills in dense metric depth, using the image for structure and the sparse returns for absolute scale — a practical hybrid of the metric-supervised (§2) and multi-view ideas that is heavily used in autonomous driving.

From posed MVS to unposed geometry foundation models

A plane-sweep volume assumes camera intrinsics and relative poses are already known. A newer direction removes that handoff: train a large feed-forward geometry model across many image collections to predict several coupled quantities at once — dense correspondences, per-view depth or point maps, camera parameters, and sometimes 3D tracks. DUSt3R predicts point maps from image pairs without requiring known cameras; MASt3R strengthens matching and multi-view alignment; VGGT processes one to many images jointly and emits cameras, depth, point maps, and tracks in a forward pass. These models turn “unposed photos → a common 3D frame” into an amortized prediction problem, then optional bundle adjustment or scene optimization can refine the result.

The widget below isolates the signal at the heart of all three regimes — stereo, MVS, and even the self-supervised photometric loss: the matching cost as a function of hypothesized depth, and what makes its valley sharp, flat, or ambiguous.

// scene intensity as a 1-D texture: sum of a few sinusoids, scaled by contrast.
// (repeated-texture mode uses a single frequency → periodic → false matches)
function tex(x){
  if (repeated) return contrast * Math.sin(0.9*x);           // periodic → aliases
  return contrast * (Math.sin(0.7*x) + 0.6*Math.sin(1.9*x+1) + 0.4*Math.sin(3.1*x));
}
// the reference sees the patch at its true position; disparity at TRUE depth:
var dTrue = f*B / Ztrue;                                      // px  (CV 05: d = fB/Z)

// sweep depth hypotheses Z; each implies a horizontal shift d(Z) = fB/Z.
// cost = mean |ref_patch - source_patch(shifted by (d(Z) - dTrue))| over the patch.
for (var i=0;i<K;i++){
  var Z = Zmin + (Zmax-Zmin)*i/(K-1);
  var d = f*B / Z;
  var shift = d - dTrue;                 // 0 exactly when Z == Ztrue → patches align
  var c = 0;
  for (var s=-W; s<=W; s++){
    var ref = tex(s);                    // reference patch sample
    var src = tex(s + shift) + noise();  // source patch, mis-aligned by `shift`
    c += Math.abs(ref - src);
  }
  cost[i] = c/(2*W+1);                   // low ONLY where shift≈0 IF texture is rich
}
// argmin over cost = recovered depth. Flat texture → cost≈const → no valley → ambiguous.

Where this points next

We can now pull 3D from ordinary images: relative or metric depth from a single frame via learned priors (§1–2), depth and ego-motion from unlabeled video by turning geometry into a photometric loss (§3), dense reconstruction from calibrated views with classical or learned MVS, and a joint geometry estimate from unposed images with modern foundation models (§4). But all remain reconstructive: they recover evidence for what was in front of cameras and must infer what was never observed. Two frontiers remain. First, generation: can we produce plausible, complete 3D — the unseen back of an object, or a whole scene from a text prompt? Second, the world moves: static reconstruction must extend to dynamic scenes. Lesson 13 takes on the modern hybrid of multiview generation, native 3D, feed-forward reconstruction, and 4D, then asks which representation and evaluation a real product can afford.

Interview prompts

• Why is monocular depth ill-posed, and what lets a network produce a depth map anyway? (§1 — projection divides by depth, so one image admits infinitely many depth maps; the network isn't inverting geometry, it's applying learned priors (perspective, texture, known sizes) — hence imperfect generalization and susceptibility to forced-perspective illusions.)
• What is the difference between relative and metric monocular depth, and must every metric model take focal length as input? (§2 — relative training factors out scale/shift; metric models learn units from metric supervision and size/scene priors. Intrinsics matter across cameras but may be fixed implicitly, supplied, canonicalized (Metric3D), or estimated; ZoeDepth commonly maps relative features to metric bins from the image alone.)
• Why train monocular depth with a scale-invariant loss, and what does Depth Anything add? (§2 — factoring out global scale+shift lets datasets at different scales/sensors be mixed, yielding robust relative depth; Depth Anything scales this via pseudo-labeling millions of unlabeled images with a teacher model.)
• Explain how self-supervised depth-from-video is trained with no ground truth. (§3 — a DepthNet and PoseNet jointly synthesize the target frame by warping a source frame (back-project via D,K, transform by T∈SE(3), reproject, sample) and minimize photometric error + edge-aware smoothness; it's lesson 01's inverse-rendering loop with a depth+pose representation.)
• What are the two main failure modes of the self-supervised photometric loss, and their fixes? (§3 — occlusion → per-pixel minimum reprojection error over multiple source frames; moving objects → auto-masking, dropping pixels the identity (no-warp) explains better, which otherwise get infinite depth.)
• Contrast classical dense MVS, learned MVS, and unposed geometry foundation models. (§4 — classical MVS already used plane sweeps/PatchMatch and photoconsistency; learned MVS learns features and cost-volume regularization from posed views; DUSt3R/MASt3R/VGGT jointly predict point maps/correspondences/cameras from unposed images, often refined geometrically. All struggle when correspondence evidence is ambiguous.)