Learned depth — monocular, self-supervised, and MVS

Everything so far needed a crutch: a depth sensor (lesson 03), multiple posed views (lesson 05 and CV 05–07), or 3D labels to supervise a network (lesson 11). None of those is an ordinary photograph. So the question this lesson forces: can we recover 3D from a single, uncalibrated, unposed image — with learning? Monocular depth is ill-posed — it is exactly the scale ambiguity of CV 04/05 and lesson 01's inverse-rendering problem, where a small near object and a large far one are indistinguishable. There is nothing to invert from one view. A network escapes only by supplying learned priors. We cover three regimes: supervised monocular depth (and why "metric" is different from "relative"), the elegant self-supervised loop that learns depth from video with no labels at all, and learned multi-view stereo when you do have several views and want dense geometry.

1 · Why monocular depth needs priors

Start from the fact CV 04 proved and lesson 01 built the whole track on: projection divides by depth. A pixel x = f·X/Z fixes a ray, not a point — scale the world point along its ray to (tX, tY, tZ) and the t cancels, so every depth along the ray lands on the same pixel. For a single image this means: every pixel's depth is a free variable. A depth map assigns one number per pixel; nothing in the image constrains it. Formally, given one image I there are infinitely many depth maps D such that back-projecting (lesson 03: p_cam = D·K^-1[u,v,1]^ᵀ) and re-projecting reproduces I. The extreme case is global scale: a photograph of a dollhouse and a photograph of a real house can be pixel-identical (this is why film miniatures work). Depth-from-one-view is not under-determined by a little — it is under-determined by an entire function.

So how can a network output a depth map at all? Not by inverting geometry — there is nothing to invert from one view, no second ray to intersect (contrast CV 05's triangulation, which needs two). It escapes the ambiguity the only way possible: by learning priors — regularities of how real scenes look, absorbed from training data. The cues it learns are exactly the ones human monocular vision uses:

• Perspective convergence. Parallel lines (rails, corridors, ceilings) meet at a vanishing point; the rate of convergence encodes surface orientation and depth.
• Texture gradients. A textured surface (gravel, grass, tiling) has finer, denser texture as it recedes — a monotone depth cue.
• Known object sizes. A person is ~1.8 m, a car ~1.5 m tall. Recognize the object, read its pixel height, and Z = f·(real size)/(pixel size) pins depth — if you know both the size and f. This is the object-size prior CV 04 flagged for monocular scale.
• Occlusion & defocus. What overlaps what gives an ordering (an occluder is nearer); the lens's depth-of-field blur is a weak metric cue.

The consequence to internalize: monocular depth is pattern-matching learned scene statistics, not solving a geometry problem. That is why it generalizes imperfectly (a scene unlike anything in training gets guessed badly) and can be actively fooled by unusual scale cues — a forced-perspective photo, a giant prop, an Ames room — where the learned priors point confidently the wrong way. The network is answering "what depth map is most likely, given scenes I've seen?", which is a probabilistic answer to a question that has no unique deterministic one.

2 · Supervised monocular depth, and metric vs relative

The supervised recipe is the obvious one: collect images each paired with a ground-truth depth map (from LiDAR, RGB-D, or a stereo rig), and train a network — an encoder-decoder or, in the modern form, a dense-prediction transformer (DPT) — to regress D = g_θ(I), minimizing a per-pixel depth loss. This works within a fixed domain. The difficulty is that ground-truth depth is scene- and sensor-specific: indoor RGB-D is in meters at one scale, driving LiDAR at another, and each dataset has its own camera. Naïvely regressing absolute meters ties the network to one sensor and one scene scale, and it fails to transfer.

The scale-invariant loss. The fix that unlocked cross-dataset training (MiDaS, then DPT) is to stop asking for absolute depth and instead compare prediction to ground truth up to an unknown global scale and shift. Concretely, before computing the error you align the prediction D to the target D^* by the best affine fit — solve for the scalar s and offset t that minimize ‖(sD + t) − D^*‖ (a 2-parameter least squares, in closed form) — and penalize only the residual:

Because the global scale and shift are factored out before scoring, the network is free to output depth in its own arbitrary units, and datasets at wildly different scales can be mixed in one training run. What the network learns is therefore relative depth — the ordering and relative structure of the scene (what is nearer than what, and by roughly how much in affine-invariant terms), not meters. This is often called ordinal or affine-invariant depth. Depth Anything pushes the same idea to scale by pseudo-labeling: a teacher model trained on the labeled data annotates millions of unlabeled internet images, and a student trains on that vastly larger, more diverse pseudo-labeled set — a semi-supervised trick that dramatically improves generalization (it is the depth-flavored cousin of the self-training in CV 14).

Getting back to meters needs the focal length. Relative depth cannot be metric because the scale-invariant loss threw scale away. To recover actual meters, the network needs the one piece of information that disambiguates size from distance: the camera's focal length f (more generally the intrinsics K). Recall from CV 04 that an object of real size X at depth Z projects to x = f·X/Z pixels — the same pixel size results from a small-near or large-far object unless you know f, which fixes the pixels↔angle conversion and lets the learned size priors resolve to a distance. So metric monocular models (ZoeDepth, Metric3D) take f (or the full K, or an equivalent FOV/canonical-camera transform) as an explicit input alongside the image. Feed the wrong focal length and a metric model's depths are systematically off by that scale factor — the same silent scale error as the wrong-K back-projection trap of lesson 03.

Which you want is set by the task, not by benchmark numbers. Relative depth suffices — and is more robust — when you only need ordering or relative structure: computational-photography effects (portrait-mode background blur, relighting), depth-aware compositing, or seeding a novel-view generator. Metric depth is mandatory the moment a real length enters the loop: robot grasping and obstacle distances, AR object placement that must sit correctly on a real table, or any measurement. Reaching for a relative-depth model where you need meters is a common and quiet mistake — it will look plausible and be unusably off in scale.

3 · Self-supervised depth from video — geometry as the loss

This is the elegant one, and the payoff of the whole track's framing. Ground-truth depth is expensive and sensor-bound; what if the only supervision were ordinary video — a moving camera, no labels, no poses? The Monodepth2 / SfM-Learner loop does exactly this, and the trick is to make geometry itself the loss. Two networks, trained jointly:

• DepthNet D = g_θ(I_t) — predicts a dense depth map of the target frame I_t.
• PoseNet T = h_φ(I_t, I_s) ∈ SE(3) — predicts the relative camera pose (lesson 02) from the target frame to a nearby source frame I_s (e.g. the previous/next frame).

The warp. With a depth for every target pixel and the relative pose, we can synthesize the target frame out of a source frame — and this synthesis chains together every operation from earlier lessons. For each target pixel p:

1. Back-project through the intrinsics and its predicted depth into a 3D point in the target camera's frame (lesson 03): P = D(p)·K^-1\,p.
2. Transform by the predicted relative pose (lesson 02) into the source camera's frame: P' = T\,P = R\,P + t.
3. Reproject that 3D point into the source image (the projection geometry of CV 04/05): p_s ∼ K\,P'.
4. Sample the source image's color at p_s (bilinearly, so the whole warp is differentiable) and paint it at p in the reconstructed target Î_t.

Do this for every pixel and you have re-rendered the target view from the source view's pixels, using only the predicted depth and pose. The single warp that maps a target pixel to its source location is:

The loss. If — and only if — the predicted depth and pose are correct, the reconstructed target Î_t matches the real target I_t. So the supervision is the photometric error between them (in practice a blend of an L1 term and structural similarity), plus an edge-aware smoothness regularizer that lets depth vary sharply only where image gradients are strong (object boundaries) and stay smooth on flat surfaces:

Both networks are trained by backprop through this differentiable warp — no depth labels, no pose labels, ever. The correct depth and pose are simply the ones that best explain how pixels actually move between consecutive frames. Read this against lesson 01's spine: the representation is (depth map + relative pose), the renderer is the warp-and-sample above, the loss is photometric, and the update is gradient descent through it. Self-supervised depth is the inverse-rendering loop with a depth+pose representation — the same loop as NeRF (lesson 09), with a far cheaper renderer.

Two wrinkles break the naïve version, and the fixes are instructive because they're pure geometry, not hacks:

• Occlusion — the per-pixel minimum reprojection error. A surface visible in the target may be hidden in a given source frame (something moved in front of it, or it left the frame). Warping into that source samples the wrong color and injects a large spurious error. The fix (Monodepth2): use multiple source frames (e.g. previous and next) and, at each target pixel, take the minimum photometric error over sources rather than the average. A pixel occluded in one frame is typically visible in another, and the min picks that good view — averaging would let the occluded frame poison the gradient.
• Moving objects — auto-masking. The whole loop assumes a static scene moved past by the camera. An object moving with the camera (a car ahead at the same speed) appears not to move between frames; the only depth that explains "no apparent motion" is infinite, so the network paints it at infinite distance — the notorious "holes of infinite depth." The fix is auto-masking: drop any pixel whose photometric error against the unwarped source (identity warp) is already lower than against the warped source. If moving nothing explains a pixel better than the predicted motion does, that pixel is not obeying the static-scene assumption, so exclude it from the loss.

4 · Learned multi-view stereo (MVS)

Monocular depth trades accuracy for needing only one image. When you do have several posed views of a scene and want dense, high-quality geometry, learned multi-view stereo is the tool. Classical multi-view stereo and SLAM (lesson 03, CV 05) recovered only sparse points — triangulated feature matches. Learned MVS produces a dense depth map per view, and the core idea is the plane sweep.

Plane sweep. Fix a reference view. Hypothesize a set of fronto-parallel depth planes at candidate depths {Z₁, …, Z_K} (usually sampled uniformly in inverse depth, since depth resolution degrades like Z² — the lesson 03 / CV 05 reason). For each hypothesized depth Z, warp every neighboring view onto that plane and back into the reference frame. Warping an image via a plane is a homography H(Z) (CV 05: a plane induces a homography between two views), parameterized by the known relative poses and intrinsics. The signal is this: at the true depth, all the warped neighbor views agree with the reference (they are all imaging the same real surface point), so the photometric variance across views is low / the matching score is high; at a wrong depth they disagree.

Cost volume. Stack the per-plane matching costs into a 3D cost volume of shape H × W × K — for each reference pixel and each depth hypothesis, a scalar cost (built in practice from the variance of deep features across the warped views, not raw pixels). This volume is noisy and has ambiguities (textureless regions, repeats). So regularize it with a 3D CNN that smooths and sharpens along the spatial and depth axes, enforcing that neighboring pixels have consistent, spatially coherent depth. Finally, collapse the depth axis to a single depth per pixel with a soft-argmin — a softmax over the (negated) cost along depth, then the cost-weighted expected depth:

The soft-argmin is differentiable (unlike a hard argmax), giving sub-plane depth precision and letting the whole thing — feature extractor, cost volume, 3D CNN — train end-to-end against ground-truth depth. This is the MVSNet family. Run it per reference view and fuse the resulting depth maps into a single point cloud or mesh by back-projecting each (lesson 03) and merging with a cross-view consistency check that rejects points not confirmed by enough neighbors — landing back on the point-cloud/mesh representations of lessons 03 and 05. A closely related task is depth completion: given a sparse LiDAR sweep and an aligned RGB image, a network fills in dense metric depth, using the image for structure and the sparse returns for absolute scale — a practical hybrid of the metric-supervised (§2) and multi-view ideas that is heavily used in autonomous driving.

The widget below isolates the signal at the heart of all three regimes — stereo, MVS, and even the self-supervised photometric loss: the matching cost as a function of hypothesized depth, and what makes its valley sharp, flat, or ambiguous.

// scene intensity as a 1-D texture: sum of a few sinusoids, scaled by contrast.
// (repeated-texture mode uses a single frequency → periodic → false matches)
function tex(x){
  if (repeated) return contrast * Math.sin(0.9*x);           // periodic → aliases
  return contrast * (Math.sin(0.7*x) + 0.6*Math.sin(1.9*x+1) + 0.4*Math.sin(3.1*x));
}
// the reference sees the patch at its true position; disparity at TRUE depth:
var dTrue = f*B / Ztrue;                                      // px  (CV 05: d = fB/Z)

// sweep depth hypotheses Z; each implies a horizontal shift d(Z) = fB/Z.
// cost = mean |ref_patch - source_patch(shifted by (d(Z) - dTrue))| over the patch.
for (var i=0;i<K;i++){
  var Z = Zmin + (Zmax-Zmin)*i/(K-1);
  var d = f*B / Z;
  var shift = d - dTrue;                 // 0 exactly when Z == Ztrue → patches align
  var c = 0;
  for (var s=-W; s<=W; s++){
    var ref = tex(s);                    // reference patch sample
    var src = tex(s + shift) + noise();  // source patch, mis-aligned by `shift`
    c += Math.abs(ref - src);
  }
  cost[i] = c/(2*W+1);                   // low ONLY where shift≈0 IF texture is rich
}
// argmin over cost = recovered depth. Flat texture → cost≈const → no valley → ambiguous.

Where this points next

We can now pull 3D from ordinary images: relative or metric depth from a single frame via learned priors (§1–2), depth and ego-motion from unlabeled video by turning geometry into a photometric loss (§3), and dense reconstruction from a handful of posed views with plane-sweep cost volumes (§4). But every method so far — classical or learned — has been reconstructive: it recovers the 3D that was there, in front of some camera, and it recovers only the parts a camera actually saw. Two frontiers remain. First, generation: can we hallucinate plausible, complete 3D — the unseen back of an object, or a whole scene from a text prompt — by lifting the 2D diffusion priors of CV 17 into 3D (Score Distillation, image-to-3D, feed-forward Gaussians)? Second, the world moves: static reconstruction must extend to dynamic scenes (4D / deformable NeRF and Gaussians). Lesson 13 — the finale — takes on generative 3D and dynamics, then steps back to the systems question every interview eventually reaches: given a real product and its constraints, which representation, which supervision, and which metric do you actually choose? Self-supervised depth and MVS are the last purely reconstructive tools; from here the track turns to creating 3D and shipping it.

Interview prompts

• Why is monocular depth ill-posed, and what lets a network produce a depth map anyway? (§1 — projection divides by depth, so one image admits infinitely many depth maps; the network isn't inverting geometry, it's applying learned priors (perspective, texture, known sizes) — hence imperfect generalization and susceptibility to forced-perspective illusions.)
• What is the difference between relative and metric monocular depth, and what does metric depth require that relative doesn't? (§2 — relative is affine-invariant ordering trained with a scale-invariant loss on mixed data; metric is actual meters and needs the camera's focal length / intrinsics as input, because f disambiguates size from distance.)
• Why train monocular depth with a scale-invariant loss, and what does Depth Anything add? (§2 — factoring out global scale+shift lets datasets at different scales/sensors be mixed, yielding robust relative depth; Depth Anything scales this via pseudo-labeling millions of unlabeled images with a teacher model.)
• Explain how self-supervised depth-from-video is trained with no ground truth. (§3 — a DepthNet and PoseNet jointly synthesize the target frame by warping a source frame (back-project via D,K, transform by T∈SE(3), reproject, sample) and minimize photometric error + edge-aware smoothness; it's lesson 01's inverse-rendering loop with a depth+pose representation.)
• What are the two main failure modes of the self-supervised photometric loss, and their fixes? (§3 — occlusion → per-pixel minimum reprojection error over multiple source frames; moving objects → auto-masking, dropping pixels the identity (no-warp) explains better, which otherwise get infinite depth.)
• Walk through learned MVS from posed images to a dense depth map, and name the shared failure. (§4 — plane-sweep neighbors via per-depth homographies, build a cost volume (low cost where views agree, i.e. at true depth), regularize with a 3D CNN, soft-argmin over depth; textureless surfaces flatten the cost curve — the same ambiguity that sinks stereo (CV 05) and the photometric loss.)