Multi-view geometry, stereo, depth & SLAM

多视图几何、立体、深度与 SLAM

Lesson 04 proved a camera throws depth away: every point on a ray (tX, tY, tZ) lands on the same pixel, so one image cannot be inverted. The fix it pointed to is more views. This lesson does the inversion — a second calibrated camera, a known baseline, and a correspondence give two rays that meet at one 3D point. (A correspondence is a pair of pixels — one in each image — that are projections of the same physical 3D point.) We derive the depth-from-disparity formula, the epipolar constraint that finds correspondences cheaply, and the scale ambiguity that monocular vision can't escape — then look at how NeRF and 3D Gaussian Splatting reconstruct whole scenes.

第 04 课证明了相机会把深度丢掉：一条射线上的每个点 (tX, tY, tZ) 都落到同一个像素上，所以单张图像无法被反演。它给出的解决方向是使用更多视图。本课就来做这个反演——第二台已标定的相机、一段已知的基线，再加上一组对应，就给出两条相交于同一个三维点的射线。（一组对应是一对像素——每张图像各一个——它们是同一个物理三维点的投影。）我们将推导深度-视差公式、能廉价找到对应的对极约束，以及单目视觉无法摆脱的尺度模糊——然后看看 NeRF 和三维高斯泼溅如何重建整个场景。

1 · Stereo: turning a pixel offset into metres

1 · 立体视觉：把像素偏移变成米

Put two identical pinhole cameras side by side, optical axes parallel, separated horizontally by a baseline B. A world point at depth Z projects into both. From lesson 04's x = f·X/Z, measure each camera's horizontal image coordinate from its own optical axis. The left camera sees the point at x_L = f·X/Z; the right camera's origin is shifted by B, so it sees the same point at x_R = f·(X − B)/Z. The disparity is the difference in those image positions:

把两台完全相同的针孔相机并排放置，光轴平行，水平方向上相隔一段基线 B。一个深度为 Z 的世界点会投影到两台相机里。根据第 04 课的 x = f·X/Z，从各自的光轴出发测量每台相机的水平图像坐标。左相机看到该点位于 x_L = f·X/Z；右相机的原点平移了 B，所以它看到同一个点位于 x_R = f·(X − B)/Z。视差就是这两个图像位置之差：

That is the workhorse equation. Read it: disparity is the apparent shift of a point between the two eyes, it is inversely proportional to depth, and inverting gives depth directly. Hold your thumb up and blink each eye — the thumb jumps a lot (big disparity, near), a far building barely moves (small disparity, far). The brain does exactly this.

这就是那条主力方程。读一读它：视差是一个点在两只眼睛之间的表观位移，它与深度成反比，反演它就直接得到深度。竖起你的大拇指，轮流眨左右眼——大拇指会跳很大一段（视差大，近），而远处的一栋楼几乎不动（视差小，远）。大脑做的正是这件事。

Worked numbers. f = 800 px, B = 0.12 m. A point with disparity d = 40 px sits at Z = 800·0.12/40 = 2.4 m. A point with d = 4 px sits at Z = 24 m. Notice the depth resolution: differentiating, |dZ| = (f·B/d²)·|Δd|, so one pixel of disparity error at d=40 moves depth by ~6 cm, but at d=4 it moves depth by ~6 m. Depth error grows like Z² — stereo is precise up close and hopeless far away, and a larger baseline B buys range at the cost of harder matching (the two views look more different). The widget below makes you feel this 1/Z relationship directly.

算个具体数。f = 800 px，B = 0.12 m。一个视差为 d = 40 px 的点位于 Z = 800·0.12/40 = 2.4 m。一个 d = 4 px 的点位于 Z = 24 m。注意深度分辨率：求导得 |dZ| = (f·B/d²)·|Δd|，所以在 d=40 处一个像素的视差误差会让深度变动约 6 cm，而在 d=4 处却会让深度变动约 6 m。深度误差随 Z² 增长——立体视觉近处精确、远处无望，而更大的基线 B 以更难的匹配为代价换来更大的量程（两个视图看起来差异更大）。下面的小部件让你直接感受这个 1/Z 关系。

Interactive · disparity ↔ depth

交互 · 视差 ↔ 深度

Two camera centres separated by baseline B, an object you drag in depth with the slider. The widget projects the object into the left and right images using x = f·X/Z, draws both rays, and reports the disparity d = f·B/Z and the depth recovered by inverting it. Push the object far away and watch disparity collapse toward zero — that vanishing signal is exactly why distant depth is unrecoverable.

两个相机中心相隔一段基线 B，还有一个你可以用滑块在深度方向上拖动的物体。小部件用 x = f·X/Z 把物体投影到左右两幅图像里，画出两条射线，并报告视差 d = f·B/Z 以及反演它所恢复出的深度。把物体推远，看着视差朝零坍缩——那个正在消失的信号，正是远处深度无法恢复的原因。

// NOTE: the prose puts the object at lateral X and the right camera at +B
//   (xR = f(X-B)/Z); here the object is centred and the cameras sit at ±B/2.
//   These are the same geometry in two frames — both give disparity d = fB/Z.
// object centred over the baseline midpoint, depth Z metres in front
const X = 0;                       // lateral offset from baseline midpoint
const xL = f * (X + B/2) / Z;      // project into left camera  (px from its axis)
const xR = f * (X - B/2) / Z;      // project into right camera
const disparity = xL - xR;         // = f*B/Z  (always positive for Z>0)
const Zrec = f * B / disparity;    // invert → recovers Z exactly
// depth resolution: one pixel of disparity error moves depth by
const dZ_per_px = f * B / (disparity*disparity);   // ∝ Z²  → far = imprecise

2 · Epipolar geometry: a point becomes a line

2 · 对极几何：一个点变成一条线

Stereo assumed we already had correspondences. Finding them is the expensive part — for each pixel in the left image, which pixel in the right is the same physical point? Searching the whole second image is O(N²) and ambiguous. Epipolar geometry shrinks the search from a 2D image to a 1D line, for free, using only the relative pose of the two cameras.

立体视觉假设我们已经有了对应。找到它们才是昂贵的部分——对于左图像里的每个像素，右图像里哪个像素是同一个物理点？搜索整幅第二张图像是 O(N²) 的，而且有歧义。对极几何仅利用两台相机的相对位姿，就免费地把搜索从一幅二维图像缩小到一条一维直线。

A note on homogeneous coordinates, because the algebra below leans on them. We write a 2-D image point (u, v) as the 3-vector x = (u, v, 1), and a 2-D line as a 3-vector (a, b, c) standing for the equation au + bv + c = 0. In this notation a point lies on a line exactly when their dot product is zero. That single trick is what lets x′ᵀFx = 0 mean "point x′ lies on line Fx" and lets the matrix product Fx literally be a line.

关于齐次坐标的一点说明，因为下面的代数会倚重它们。我们把一个二维图像点 (u, v) 写成三维向量 x = (u, v, 1)，把一条二维直线写成一个三维向量 (a, b, c)，代表方程 au + bv + c = 0。在这套记号下，一个点落在一条直线上，当且仅当它们的点积为零。正是这一个小技巧，让 x′ᵀFx = 0 得以表示"点 x′ 落在直线 Fx 上"，并让矩阵乘积 Fx 字面上就是一条直线。

Here's the picture. A pixel x in the left image back-projects to a ray through the left optical centre. The matching point lies somewhere on that ray, at unknown depth. Now project that entire ray into the right image: a 3D line projects to a 2D line. So the match for x must lie on a specific line in the right image — its epipolar line. All such lines pass through the epipole, the image of the other camera's centre. Two calibrated cameras give a 3×3 matrix encoding this:

这幅图景是这样的。左图像里的一个像素 x 反投影成一条穿过左光心的射线。匹配点就在这条射线上的某处，深度未知。现在把整条射线投影到右图像里：一条三维直线投影成一条二维直线。所以 x 的匹配点必定落在右图像里一条特定的直线上——它的对极线。所有这样的线都通过对极点，也就是另一台相机中心的像。两台已标定的相机给出一个 3×3 矩阵来编码这一点：

Read x′ᵀFx = 0 as: "the correspondence x′ lies on the line Fx." The product Fx is a 3-vector that is the epipolar line (a line [a,b,c] in homogeneous form means au+bv+c=0), and the constraint says x′ sits on it. The fundamental matrix F works in raw pixels with unknown intrinsics; the essential matrix E works on calibrated, intrinsics-removed rays x̂ = K⁻¹x. They're related by exactly the intrinsics from lesson 04: F = K′⁻ᵀ E K⁻¹. This is what we meant when we said K converts one into the other.

把 x′ᵀFx = 0 读作："对应点 x′ 落在直线 Fx 上。"乘积 Fx 是一个三维向量，它就是那条对极线（齐次形式下一条直线 [a,b,c] 表示 au+bv+c=0），而这个约束说的是 x′ 落在它上面。基础矩阵 F 在内参未知的原始像素上工作；本质矩阵 E 在已标定、已去除内参的射线 x̂ = K⁻¹x 上工作。它们正是通过第 04 课的内参联系起来的：F = K′⁻ᵀ E K⁻¹。这就是我们说 K 把一个转换成另一个时的含义。

3 · Essential matrix, scale ambiguity, triangulation, SfM & SLAM

3 · 本质矩阵、尺度模糊、三角化、SfM 与 SLAM

The essential matrix gives direction, not distance. Decompose E and you recover the relative rotation R and translation t between the two cameras — but only t's direction, never its length. The reason is structural: E = [t]_×R is built from a cross product. E falls out of one fact: the two rays and the baseline t all lie in one plane (they form a triangle), so the scalar triple product x̂′·(t × Rx̂) = 0; writing the cross product as the matrix [t]_× turns this into x̂′ᵀ([t]_×R)x̂ = 0, and E = [t]_×R. Here [t]_× is the skew-symmetric matrix that implements the cross product — it is the 3×3 matrix for which t × v = [t]_× v for any vector v, packaging "cross with t" as a matrix multiply. Because [t]_× is linear in t, scaling t by any factor k scales E by exactly k, which doesn't change x′ᵀEx = 0 — so the scale of t is unrecoverable. Geometrically, a scene twice as big seen from a baseline twice as long produces identical images. This is the monocular scale ambiguity: from images alone you reconstruct the world up to an unknown global scale. To pin metres you need an external anchor — a known baseline (stereo), a known object size, IMU/wheel odometry, or a depth sensor. It's the same "depth divides, depth destroyed" loss from lesson 04, now surfacing as one missing number for the whole reconstruction. One subtlety when you actually decompose E: the factorization is not unique — it yields four candidate (R, t) solutions (two possible rotations × two sign choices for the translation direction). You disambiguate with the cheirality check: triangulate a point under each candidate and keep the one for which the reconstructed 3D point lies in front of both cameras (positive depth) — only one of the four does.

本质矩阵给出的是方向，而非距离。分解 E，你就能恢复出两台相机之间的相对旋转 R 和平移 t——但只有 t 的方向，永远得不到它的长度。原因是结构性的：E = [t]_×R 是由一个叉积构造出来的。E 源于一个事实：两条射线和基线 t 都落在同一个平面内（它们构成一个三角形），所以标量三重积 x̂′·(t × Rx̂) = 0；把叉积写成矩阵 [t]_× 就把它变成 x̂′ᵀ([t]_×R)x̂ = 0，于是 E = [t]_×R。这里 [t]_× 是实现叉积的反对称矩阵——它是那个使得对任意向量 v 都有 t × v = [t]_× v 的 3×3 矩阵，把"与 t 做叉乘"打包成一次矩阵乘法。由于 [t]_× 关于 t 是线性的，把 t 乘以任意因子 k 就恰好把 E 乘以 k，而这并不改变 x′ᵀEx = 0——所以 t 的尺度是无法恢复的。从几何上看，一个大一倍的场景从长一倍的基线上看过去，产生的图像完全相同。这就是单目尺度模糊：仅凭图像，你重建出的世界只能确定到一个未知的全局尺度。要钉死米制，你需要一个外部锚——已知的基线（立体视觉）、已知的物体尺寸、IMU/轮式里程计，或一个深度传感器。这与第 04 课"深度做了除法、深度就被摧毁"是同一个损失，如今它以整个重建缺失的一个数字的形式浮现出来。真正分解 E 时有一个微妙之处：这个分解并不唯一——它给出四个候选的 (R, t) 解（两种可能的旋转 × 平移方向的两种符号选择）。你用手性检验（cheirality check）来消除歧义：在每个候选下三角化一个点，保留那个重建出的三维点位于两台相机前方（深度为正）的解——四个里只有一个满足。

Triangulation is the actual inversion: given two known camera poses and a correspondence, the 3D point is where the two back-projected rays meet. With noise the rays almost never intersect exactly, so you don't look for a true crossing — you solve a small least-squares problem (the DLT, direct linear transform: stack the two projection equations and find the 3D point that best satisfies them) for the point minimizing reprojection error, the point whose projections land closest to both observed pixels. Repeat over all correspondences to get a point cloud.

三角化才是真正的反演：给定两个已知的相机位姿和一组对应，三维点就是两条反投影射线相交之处。有噪声时两条射线几乎从不精确相交，所以你并不去寻找一个真正的交点——你求解一个小的最小二乘问题（DLT，直接线性变换：把两个投影方程堆叠起来，找出最好地满足它们的那个三维点），得到使重投影误差最小的点，也就是其投影最靠近两个观测像素的点。对所有对应重复这一过程，就得到一个点云。

The table below names the handful of geometric tools that recur for the rest of the lesson. Two are worth defining in prose before you meet them in the rows. A homography H is a 3×3 matrix that maps points in one image directly to another — valid only when the scene is planar or the camera purely rotates, so there is no parallax (the apparent shift of a point between viewpoints — the same effect as stereo disparity in §1) to recover depth from. PnP stands for Perspective-n-Point: given n known 3D points and where they land in the image, solve for the camera's pose.

下表列出了本课其余部分反复出现的少数几个几何工具。其中两个值得在行中遇到它们之前先用文字定义一下。一个单应矩阵 H 是一个 3×3 矩阵，它把一幅图像里的点直接映射到另一幅——仅当场景是平面的、或相机做纯旋转时才成立，因为这时没有视差（一个点在不同视点之间的表观位移——与 §1 里的立体视差是同一种效应）可供恢复深度。PnP 代表 Perspective-n-Point（透视 n 点）：给定 n 个已知的三维点以及它们落在图像中的位置，求解相机的位姿。

Structure-from-Motion (SfM) scales this up: match features across many images, estimate relative poses pairwise, triangulate points, and run bundle adjustment — a big nonlinear least-squares that jointly nudges every camera pose and every 3D point to minimize total reprojection error. It scales to thousands of cameras and millions of points only because its Jacobian is sparse: each observation touches exactly one camera and one point, so the system has a block structure that the Schur complement (and sparse linear solvers) exploit to avoid the cost of a dense solve. SLAM (Simultaneous Localization And Mapping) does the same incrementally and in real time on a moving camera: track features, estimate the new pose (PnP), extend the map, detect loop closures (recognizing a previously seen place to correct accumulated drift), and bundle-adjust a sliding window. PnP vs triangulation: use PnP when the map (3D points) is known and you want the camera pose; use triangulation when the poses are known and you want new 3D points. SLAM alternates between the two every frame.

运动恢复结构（SfM）把这一切扩大：在许多图像间匹配特征，两两估计相对位姿，三角化点，再运行光束平差——一个大型非线性最小二乘，联合微调每一个相机位姿和每一个三维点，使总重投影误差最小。它之所以能扩展到数千台相机和数百万个点，只是因为它的雅可比矩阵是稀疏的：每个观测恰好只涉及一台相机和一个点，于是系统具有块状结构，Schur 补（以及稀疏线性求解器）利用这一结构来避免稠密求解的开销。SLAM（同时定位与建图，Simultaneous Localization And Mapping）在一台运动的相机上增量地、实时地做同样的事：跟踪特征、估计新位姿（PnP）、扩展地图、检测回环（认出一个先前见过的地方以纠正累积的漂移），并对一个滑动窗口做光束平差。PnP 与三角化之别：当地图（三维点）已知而你想要相机位姿时，用 PnP；当位姿已知而你想要新的三维点时，用三角化。SLAM 每一帧都在两者之间交替。

SLAM and SfM are brittle exactly where their assumptions break: textureless surfaces (no features to match — the hallway problem), repeated patterns (false matches that pass the ratio test), dynamic objects (violate the static-scene assumption baked into triangulation), motion blur, rolling shutter, and insufficient parallax (pure rotation gives no triangulation — depth is unobservable, same as monocular).

SLAM 和 SfM 恰恰在它们的假设被打破的地方变得脆弱：无纹理表面（没有特征可匹配——走廊问题）、重复图案（通过了比值检验的错误匹配）、动态物体（违反了三角化内建的静态场景假设）、运动模糊、卷帘快门，以及视差不足（纯旋转无法三角化——深度不可观测，与单目相同）。

4 · Modern 3D: NeRF vs 3D Gaussian Splatting

4 · 现代三维：NeRF 与三维高斯泼溅

Classical pipelines output sparse points or meshes. Two recent methods instead reconstruct a continuous scene you can render from new viewpoints — they still pay the same geometric bill (you need calibrated poses, usually from SfM), but model appearance far more richly.

经典流水线输出稀疏的点或网格。而最近的两种方法则重建出一个可以从新视点渲染的连续场景——它们仍然要付同样的几何账（你需要已标定的位姿，通常来自 SfM），但对外观的建模丰富得多。

NeRF (Neural Radiance Field) represents a scene as a neural network: feed it a 3D position and viewing direction, it returns color and volume density. To render a pixel you march a ray through the scene, sample many points along it, query the network at each, and integrate color weighted by density (volumetric rendering — concretely: walk along the camera ray, sample color and density at points spaced down it, and alpha-composite those samples front-to-back into the single color of that one pixel). It's photorealistic but slow — hundreds of network queries per pixel means minutes-to-hours of training and seconds per frame to render.

NeRF（神经辐射场，Neural Radiance Field）把一个场景表示成一个神经网络：给它一个三维位置和一个观察方向，它返回颜色和体密度。要渲染一个像素，你让一条射线穿过场景，沿途采样许多点，在每个点查询网络，再把按密度加权的颜色积分起来（体渲染——具体来说：沿相机射线前进，在射线上间隔分布的点处采样颜色和密度，再把这些采样从前到后做 alpha 合成，得到这一个像素的单一颜色）。它逼真但缓慢——每个像素上百次的网络查询，意味着训练要数分钟到数小时、渲染每帧要数秒。

3D Gaussian Splatting (3DGS) represents the scene explicitly as millions of little 3D Gaussian blobs, each with a position, covariance (shape), color, and opacity. To render, you project ("splat") each Gaussian onto the image plane and blend them front-to-back with the GPU rasterizer — the same fixed-function pipeline that draws triangles in games. Why 3DGS is real-time and NeRF isn't: NeRF does volumetric ray-marching with a neural-network query at every sample point along every ray (expensive, implicit); 3DGS does a single rasterization pass over explicit primitives with no per-sample network call (cheap, parallel, hardware-accelerated). Explicit primitives + rasterization beats implicit field + ray-marching by orders of magnitude, which is why 3DGS renders at 100+ FPS where NeRF crawled.

三维高斯泼溅（3D Gaussian Splatting，3DGS）把场景显式地表示成数百万个小小的三维高斯团，每个都带有一个位置、协方差（形状）、颜色和不透明度。要渲染，你把每个高斯投影（"泼溅"）到像平面上，再用 GPU 光栅器把它们从前到后混合——就是游戏里画三角形用的那套固定功能流水线。为什么 3DGS 是实时的而 NeRF 不是：NeRF 做的是体渲染的射线步进，沿每条射线的每个采样点都要查询一次神经网络（昂贵、隐式）；3DGS 只在显式图元上做一趟光栅化，没有逐采样的网络调用（廉价、并行、硬件加速）。显式图元 + 光栅化以数量级的优势胜过隐式场 + 射线步进，这正是为什么 3DGS 能以 100+ FPS 渲染，而 NeRF 却慢如爬行。

Where this points next

这将指向何处

We've now built the entire classical 3D vision pipeline: detect features (03), model the camera (04), use epipolar geometry to verify matches and rectify, triangulate to depth, and bundle-adjust into a map (05). Every stage is hand-engineered and brittle — it fails on texture, dynamics, repeats, and bad calibration, and each fix is another hand-coded heuristic. That fragility is the motivation for the rest of the track. Lesson 06 asks the question that reframes everything: instead of hand-designing the corner detectors of lesson 03 and stacking geometric rules, can a network learn the features and the task end-to-end, directly from data? The convolutional neural network is the architecture that makes that practical — and, strikingly, its early layers rediscover the edge and corner detectors we built by hand, while learning the rest.

现在我们已经搭出了整套经典的三维视觉流水线：检测特征（03）、给相机建模（04）、用对极几何验证匹配并做校正、三角化出深度，再通过光束平差得到一张地图（05）。每一个阶段都是手工设计的、脆弱的——它在纹理、动态、重复图案和糟糕的标定上失效，而每一个修补又是另一条手写的启发式规则。这种脆弱性正是本课其余部分的动机。第 06 课提出了那个重新定义一切的问题：与其手工设计第 03 课的角点检测器、再堆叠几何规则，一个网络能否直接从数据中端到端地学到特征和任务？卷积神经网络正是让这变得可行的架构——而且引人注目的是，它的早期层重新发现了我们手工构造的那些边缘和角点检测器，同时把其余的部分也学了出来。

Interview prompts

面试题

• Why is depth ambiguous from one image? (Projection is many-to-one along the ray; t cancels in f·tX/tZ.)为什么单张图像的深度是有歧义的？（投影沿射线是多对一的；t 在 f·tX/tZ 中被约掉了。）
• How does stereo baseline affect depth accuracy? (Bigger B → larger disparity → finer depth, but harder matching; error scales as Z²/B.)立体基线如何影响深度精度？（更大的 B → 更大的视差 → 更精细的深度，但匹配更难；误差按 Z²/B 变化。）
• Difference between essential and fundamental matrix? (F in pixels/unknown intrinsics; E on calibrated rays; F = K′⁻ᵀEK⁻¹.)本质矩阵与基础矩阵的区别？（F 作用于像素/内参未知；E 作用于已标定的射线；F = K′⁻ᵀEK⁻¹。）
• What does x′ᵀFx = 0 mean? (The match x′ lies on the epipolar line Fx — search reduces to a line.)x′ᵀFx = 0 意味着什么？（匹配点 x′ 落在对极线 Fx 上——搜索被降到一条线上。）
• When PnP vs triangulation? (PnP: known map, want pose. Triangulation: known poses, want points.)何时用 PnP、何时用三角化？（PnP：地图已知，想要位姿。三角化：位姿已知，想要点。）
• When is a homography the right model? (Planar scene or pure camera rotation.)何时单应矩阵是正确的模型？（平面场景或纯相机旋转。）
• Why do SLAM systems fail in textureless hallways or dynamic scenes? (No features to match; moving objects break the static-scene/triangulation assumption.)为什么 SLAM 系统在无纹理的走廊或动态场景中会失效？（没有特征可匹配；运动物体破坏了静态场景/三角化假设。）
• Why is 3D Gaussian Splatting real-time when NeRF isn't? (Explicit primitives + rasterization vs implicit field + per-sample neural ray-marching.)为什么三维高斯泼溅是实时的而 NeRF 不是？（显式图元 + 光栅化，对比隐式场 + 逐采样的神经射线步进。）