Cameras, projection, calibration & distortion

相机、投影、标定与畸变

Lesson 03 turned two images into a set of 2D correspondences — pairs of pixels (u, v) ↔ (u′, v′) we trust point at the same physical thing. But a pixel is not a point in the world; it is the shadow a 3D point cast onto a sensor. Before we can climb back from those correspondences to 3D (lesson 05), we need the exact map that did the casting: the camera model. This lesson derives it from a hole in a wall.

第 03 课把两幅图像变成了一组二维对应关系——一对对我们相信指向同一个物理实体的像素 (u, v) ↔ (u′, v′)。但像素并不是世界中的一个点；它是一个三维点投在传感器上的影子。在我们能从这些对应关系爬回三维（第 05 课）之前，我们需要那张完成投射的精确映射：相机模型。本课将从墙上的一个孔把它推导出来。

1 · From a pinhole to x = f·X/Z

1 · 从针孔到 x = f·X/Z

Imagine a sealed box with a tiny hole in the front face. Light from a point in the world travels in straight rays; only the ray heading exactly at the hole gets through, and it lands on the back wall — the image plane. Put the camera's coordinate origin at the hole (the optical center), let the Z axis point out along the direction the camera looks (the optical axis), and let X, Y span the image directions. A world point sits at (X, Y, Z) in this frame, where Z is its distance in front of the camera (its depth).

想象一个密封的盒子，正面有一个极小的孔。来自世界中某个点的光沿直线传播；只有正对着这个孔的那条光线能穿过去，落在后壁上——也就是像平面。把相机坐标系的原点放在孔处（光心），让 Z 轴沿相机观看的方向指向外（光轴），并让 X、Y 张成图像方向。一个世界点在这个坐标系里位于 (X, Y, Z)，其中 Z 是它在相机前方的距离（它的深度）。

To avoid the upside-down real image, we use the standard trick of placing a virtual image plane in front of the hole at distance f (the focal length) along Z. Now draw the ray from the world point (X, Z) through the origin to where it crosses the plane Z = f, and look at it from the side (the X–Z plane):

为了避免那个上下颠倒的实像，我们采用标准技巧：把一个虚拟像平面放在孔前方、沿 Z 轴距离为 f（焦距）处。现在从世界点 (X, Z) 过原点画出光线，一直画到它穿过平面 Z = f 的地方，并从侧面（X–Z 平面）来看它：

The big triangle (optical center → world point → its foot on the axis) and the small triangle (optical center → image point → its foot on the axis) share the angle at O and both have a right angle. They are similar, so corresponding sides are in equal ratio. The small triangle has horizontal run f and rise x; the big triangle has run Z and rise X:

大三角形（光心 → 世界点 → 它在轴上的垂足）和小三角形（光心 → 像点 → 它在轴上的垂足）在 O 处共享同一个角，且都有一个直角。它们是相似的，所以对应边成等比。小三角形的水平边长为 f、竖直边长为 x；大三角形的水平边长为 Z、竖直边长为 X：

That is the entire perspective camera. Everything else is bookkeeping. Two consequences fall straight out and are worth saying in words:

这就是透视相机的全部。其余一切都只是记账。有两个推论直接落地，值得用文字讲清楚：

• Depth divides. The same physical width X at twice the distance Z projects to half the image size — this single 1/Z is why far things look small, why parallel rails appear to meet, and (lesson 05) why depth is what we have to recover.深度做除法。同样的物理宽度 X 放在两倍的距离 Z 上，投影到图像上的尺寸减半——正是这个 1/Z 让远处的东西看起来小、让平行的铁轨看起来交汇于一点，并且（第 05 课）让深度成了我们必须去恢复的东西。
• Depth is destroyed. Every point on the ray (tX, tY, tZ) for any t>0 projects to the same (x, y), because the t cancels. A pixel pins down a ray, not a point. The map from 3D to 2D is many-to-one, so it cannot be inverted from one image — exactly the loss lesson 05 spends two views undoing.深度被摧毁。对任意 t>0，射线 (tX, tY, tZ) 上的每一个点都投影到同一个 (x, y)，因为 t 被约掉了。一个像素锁定的是一条射线，而不是一个点。从三维到二维的映射是多对一的，所以无法从单幅图像把它反演出来——这正是第 05 课要用两个视角去撤销的那份损失。

Worked number. A camera with f = 800 px images a person X = 1.8 m tall standing Z = 6 m away. Their head-to-foot extent projects to 800 · 1.8 / 6 = 240 px. Walk them to Z = 12 m and it halves to 120 px. The pixel size carries depth information — if and only if you know the real size, which is the object-size prior lesson 05 leans on for monocular scale.

算个具体数。一台 f = 800 px 的相机拍摄一个身高 X = 1.8 m、站在 Z = 6 m 外的人。他从头到脚的高度投影为 800 · 1.8 / 6 = 240 px。让他走到 Z = 12 m，高度减半为 120 px。像素尺寸携带了深度信息——当且仅当你知道真实尺寸时才成立，这正是第 05 课在单目尺度问题上所倚重的物体尺寸先验。

2 · Pixels, homogeneous coordinates, and the matrix K

2 · 像素、齐次坐标与矩阵 K

The equation x = f·X/Z gives a position in metric units on the image plane, measured from where the optical axis pierces it. A real sensor reports something different: integer pixel coordinates (u, v) counted from the top-left corner. Three adjustments bridge the gap.

方程 x = f·X/Z 给出的是像平面上以公制单位表示的位置，从光轴穿过它的那一点算起。而真实的传感器报告的是另一种东西：从左上角数起的整数像素坐标 (u, v)。三处调整弥合了这道鸿沟。

(a) Metric → pixels. A pixel has a physical width and height on the sensor; let there be m_x pixels per metre horizontally and m_y vertically. Folding these into the focal length gives focal lengths in pixels: f_x = m_x·f and f_y = m_y·f. They differ only if pixels aren't square. (b) Re-origin. The optical axis hits the image not at the corner but at the principal point (c_x, c_y), usually near the image centre; we add it as an offset. (c) Skew. A tiny term γ (often 0) corrects for sensor axes that aren't perfectly perpendicular — we name it γ, not s, to keep it distinct from the homogeneous scale below. Putting it together:

(a) 公制 → 像素。一个像素在传感器上有物理的宽和高；设水平方向每米有 m_x 个像素、竖直方向有 m_y 个。把它们并入焦距，就得到以像素为单位的焦距：f_x = m_x·f 和 f_y = m_y·f。只有当像素不是正方形时，二者才不同。(b) 重设原点。光轴打到图像上不是在角上，而是在主点 (c_x, c_y)，通常靠近图像中心；我们把它作为一个偏移量加进去。(c) 倾斜。一个微小项 γ（通常为 0）用来校正并非完全垂直的传感器坐标轴——我们把它命名为 γ 而不是 s，以便与下面的齐次尺度区分开。合到一起：

In practice γ = 0 for any modern sensor — its pixels are square and its axes orthogonal — so the γ·Y/Z term vanishes; γ is carried only so K keeps the general upper-triangular form. (Accordingly the worked projection below uses γ = 0.)

实践中，任何现代传感器都有 γ = 0——它的像素是正方形、坐标轴正交——所以 γ·Y/Z 项消失了；保留 γ 只是为了让 K 保持一般的上三角形式。（相应地，下面那个投影算例用的是 γ = 0。）

This is correct but ugly: the division by Z makes it nonlinear, so it can't be a matrix multiply — and we badly want a matrix, because chains of matrices compose. Homogeneous coordinates are the fix. The idea: represent a 2D point by a 3-vector [u, v, 1]ᵀ, and agree that scaling the whole vector by any nonzero number names the same point — [u, v, 1]ᵀ and [su, sv, s]ᵀ are the same pixel, and you recover the pixel by dividing through by the last entry. That deferred division is exactly the perspective divide by Z, now hidden inside the representation. So we write

这是对的，却很丑：对 Z 的除法让它变成非线性，所以它不能是一次矩阵乘法——而我们非常想要一个矩阵，因为一串矩阵可以复合。齐次坐标正是解药。思路是：用一个三维向量 [u, v, 1]ᵀ 来表示一个二维点，并约定把整个向量乘以任意非零数都表示同一个点——[u, v, 1]ᵀ 和 [su, sv, s]ᵀ 是同一个像素，你只要除以最后一个元素就能还原出该像素。这个被推迟的除法，正是对 Z 的透视除法，如今被藏进了这种表示里。于是我们写下

Read the right side bottom-up: the third row of K[X,Y,Z]ᵀ is just Z, so the unknown scale s on the left equals Z. Dividing the first two rows by that third row is the perspective divide, and it reproduces (a)–(c) exactly. So s is the depth Z — the thing the image throws away — and the homogeneous 1 is the placeholder that lets a linear matrix encode a nonlinear divide.

自下而上地读右边：K[X,Y,Z]ᵀ 的第三行就是 Z，所以左边那个未知的尺度 s 等于 Z。用第三行去除前两行就是透视除法，它精确地重现了 (a)–(c)。所以 s 就是深度 Z——正是图像丢掉的那个量——而齐次的那个 1 是一个占位符，它让一个线性矩阵得以编码一次非线性的除法。

The full pipeline. A point lives in the world frame, not the camera frame. A rigid motion — rotation R (3×3) and translation t (3×1), together the extrinsics — moves it into the camera frame: X_cam = R·X_world + t. Stacking that into homogeneous form too, the whole camera is one product:

完整的流水线。一个点活在世界坐标系里，而不是相机坐标系里。一次刚体运动——旋转 R（3×3）与平移 t（3×1），合起来就是外参——把它搬进相机坐标系：X_cam = R·X_world + t。把这一步也堆进齐次形式，整台相机就是一个乘积：

Worked number — multiply it out once. Take a concrete intrinsics matrix and put the camera at the world origin looking straight down +Z, so the extrinsics are trivial (R = I, t = 0):

算个具体数——把它乘出来一遍。取一个具体的内参矩阵，把相机放在世界原点沿 +Z 正对着看，于是外参是平凡的（R = I、t = 0）：

Project the single world point (X,Y,Z) = (0.5, 0.3, 4) metres. Because [R|t] is identity-with-zero-translation, the camera-frame point is the same (0.5, 0.3, 4), and we just multiply K by it:

投影这一个世界点 (X,Y,Z) = (0.5, 0.3, 4) 米。因为 [R|t] 是单位阵加零平移，相机坐标系里的点还是同一个 (0.5, 0.3, 4)，我们只需用 K 去乘它：

The third entry is s = 4 — and indeed s = Z, as promised. Now do the perspective divide (divide the first two entries by the third) to read off the pixel:

第三个元素是 s = 4——果然 s = Z，正如所承诺的。现在做透视除法（把前两个元素除以第三个）来读出像素：

Sanity-check against the bare similar-triangles form: u = f_x·X/Z + c_x = 800·(0.5/4) + 320 = 100 + 320 = 420 ✓, and v = 800·(0.3/4) + 240 = 60 + 240 = 300 ✓. The homogeneous machinery and the raw formula agree — the matrix form is just the same arithmetic packaged so it composes. Push the point to Z = 8 and the offset from the principal point halves (u = 800·(0.5/8)+320 = 370, closer to c_x=320): far things shrink toward the center. The widget below lets you slide X, Y, Z, f, c_x and watch every number in this calculation move.

用赤裸的相似三角形形式做个校验：u = f_x·X/Z + c_x = 800·(0.5/4) + 320 = 100 + 320 = 420 ✓，以及 v = 800·(0.3/4) + 240 = 60 + 240 = 300 ✓。齐次这套机器和原始公式一致——矩阵形式只是同样的算术被打包成可以复合的样子。把这个点推到 Z = 8，它相对主点的偏移减半（u = 800·(0.5/8)+320 = 370，更靠近 c_x=320）：远处的东西朝中心收缩。下面的小部件让你滑动 X, Y, Z, f, c_x，看着这套计算里的每一个数字一起动。

2.5 · Interactive · the pinhole projector

2.5 · 交互 · 针孔投影器

Projection is the most spatial idea in the track, so here it is as a live picture. Slide the world point's X, Y, Z and the camera's focal length f and principal point c_x, and watch the full pipeline u = f·X/Z + c_x, v = f·Y/Z + c_y compute and land a pixel on the image plane. Crank Z up and the projected point marches toward the principal point and the object shrinks — the 1/Z law, visible.

投影是这门课里最具空间感的思想，所以这里把它做成一张活的图。滑动世界点的 X, Y, Z 以及相机的焦距 f 和主点 c_x，看着完整的流水线 u = f·X/Z + c_x、v = f·Y/Z + c_y 计算出来，并把一个像素落在像平面上。把 Z 拧大，投影点就朝主点行进、物体随之缩小——1/Z 定律，看得见。

// K = [[fx,0,cx],[0,fy,cy],[0,0,1]];  camera at origin, R = I, t = 0
function project(X, Y, Z, fx, fy, cx, cy){
  // homogeneous result BEFORE the perspective divide: s*[u,v,1] = K*[X,Y,Z]
  const su = fx*X + cx*Z;      // first row of K times [X,Y,Z]
  const sv = fy*Y + cy*Z;      // second row
  const s  = Z;                // third row of K is [0,0,1] -> s = Z
  return { su, sv, s,
           u: su / s,          // perspective divide
           v: sv / s };        // u = fx*X/Z + cx,  v = fy*Y/Z + cy
}

3 · How K changes under crop and resize

3 · K 在裁剪与缩放下如何变化

This is the interview trap, and the answer follows mechanically from "K maps camera rays to pixel indices." Any operation that renumbers pixels must be reflected in K, or every downstream geometry computation silently uses the wrong camera.

这就是那个面试陷阱，而答案机械地从"K 把相机射线映射到像素索引"推出来。任何给像素重新编号的操作都必须反映到 K 里，否则下游的每一次几何计算都会悄悄地用错相机。

• Crop by removing (a, b) pixels from the left and top. The optical axis still pierces the same physical spot, but that spot is now a columns and b rows closer to the new origin. Only the principal point moves: c_x ← c_x − a, c_y ← c_y − b. Focal lengths are unchanged — cropping doesn't magnify, it just discards border.裁剪：从左边和上边各去掉 (a, b) 个像素。光轴仍然穿过同一个物理点，但那个点如今离新原点近了 a 列、b 行。只有主点移动：c_x ← c_x − a、c_y ← c_y − b。焦距不变——裁剪不会放大，它只是丢掉边缘。
• Resize by scale factors α in x and β in y (e.g. α = β = 0.5 to halve resolution). Everything measured in pixels scales: f_x ← α f_x, c_x ← α c_x, f_y ← β f_y, c_y ← β c_y. Note focal length in pixels does change here, because a pixel now covers twice the field — yet the field of view does not.缩放：在 x 方向用比例因子 α、在 y 方向用 β（例如 α = β = 0.5 把分辨率减半）。所有以像素度量的量都随之缩放：f_x ← α f_x、c_x ← α c_x、f_y ← β f_y、c_y ← β c_y。注意以像素为单位的焦距在这里确实变了，因为现在一个像素覆盖了两倍的视野——然而视场角并不变。

Worked number. Start with K having f_x=f_y=800, c_x=640, c_y=360 on a 1280×720 image. Crop 100 px off the left → c_x=540. Then downscale by ½ → f_x=f_y=400, c_x=270, c_y=180. Skip either step and a network trained on full-res images will localize objects in the wrong place, and any triangulation will be biased. This is the same train/serve preprocessing-parity contract lesson 01 raised — geometry just makes the bug numeric instead of cosmetic.

算个具体数。从一个 K 出发，它在 1280×720 图像上有 f_x=f_y=800, c_x=640, c_y=360。从左边裁掉 100 px → c_x=540。再缩小到 ½ → f_x=f_y=400, c_x=270, c_y=180。漏掉任何一步，一个在全分辨率图像上训练的网络都会把物体定位到错误的位置，任何三角化也都会有偏差。这与第 01 课提出的训练/服务预处理一致性契约是同一回事——几何只是把这个 bug 从表面问题变成了数值问题。

4 · Field of view, distortion, and calibration

4 · 视场角、畸变与标定

FOV ↔ focal length. The horizontal field of view is the angle the sensor subtends through the pinhole. With image width W pixels, the half-angle satisfies tan(FOV_x/2) = (W/2) / f_x, so

视场角 ↔ 焦距。水平视场角就是传感器透过针孔所张的角度。设图像宽度为 W 像素，则半角满足 tan(FOV_x/2) = (W/2) / f_x，于是

Long focal length (large f_x) → small angle → telephoto, narrow and "flattened." Short focal length → wide angle, lots of scene, strong perspective. Worked number: W=1280, f_x=800 gives 2·arctan(640/800) = 2·38.7° ≈ 77° — a normal lens. Halve f_x to 400 and FOV jumps to 2·arctan(1.6) ≈ 116°, a wide angle. This is why "what's the focal length?" and "what's the FOV?" are the same question once you know W.

长焦距（大 f_x）→ 小角度 → 长焦，视野窄且"被压平"。短焦距 → 广角，场景多，透视强烈。算个具体数：W=1280、f_x=800 给出 2·arctan(640/800) = 2·38.7° ≈ 77°——一枚标准镜头。把 f_x 减半到 400，FOV 就跳到 2·arctan(1.6) ≈ 116°，成了广角。这就是为什么一旦你知道 W，"焦距是多少？"和"视场角是多少？"就是同一个问题。

Distortion. A real lens is not an ideal pinhole. Radial distortion bends straight lines into arcs — bowing outward (barrel, typical of wide lenses) or inward (pincushion) — and grows with distance from the centre; it's modeled by a polynomial in radius, r' = r(1 + k₁r² + k₂r⁴ + …). Worked number: take a barrel lens with k₁ = −0.2 (negative k₁ pulls inward) and a point at normalized radius r = 0.8; then r' = 0.8·(1 + (−0.2)(0.8²)) = 0.8·(1 − 0.2·0.64) = 0.8·(1 − 0.128) = 0.8·0.872 = 0.698 — the corner is pulled inward by about (0.8−0.698)/0.8 ≈ 13%, which is exactly the inward bow that makes straight lines near the frame edge curve. Tangential distortion appears when the lens and sensor planes aren't perfectly parallel, adding a small sideways shift (params p₁, p₂). Distortion acts before the linear K step, on the normalized ray, so the clean x=f·X/Z picture only holds after you undistort.

畸变。真实的镜头不是理想的针孔。径向畸变把直线弯成弧线——向外鼓（桶形，广角镜头的典型）或向内凹（枕形）——且随着与中心距离的增大而增强；它由一个关于半径的多项式来建模，r' = r(1 + k₁r² + k₂r⁴ + …)。算个具体数：取一枚桶形镜头，k₁ = −0.2（负的 k₁ 向内拉）以及一个位于归一化半径 r = 0.8 处的点；则 r' = 0.8·(1 + (−0.2)(0.8²)) = 0.8·(1 − 0.2·0.64) = 0.8·(1 − 0.128) = 0.8·0.872 = 0.698——这个角点被向内拉了约 (0.8−0.698)/0.8 ≈ 13%，这正是让画面边缘附近的直线弯曲的那种向内内凹。切向畸变出现在镜头平面与传感器平面不完全平行时，会添加一个微小的侧向偏移（参数 p₁, p₂）。畸变作用在线性 K 步骤之前、作用在归一化射线上，所以那幅干净的 x=f·X/Z 图景只有在你去畸变之后才成立。

Calibration is the procedure that recovers K and the distortion coefficients. The standard method shows the camera a known pattern (a checkerboard) from many angles: each view gives a batch of known 3D corner positions paired with their detected pixels, and you solve for the intrinsics, the per-view extrinsics, and distortion that best reproject the corners — minimizing reprojection error, the pixel gap between where a known 3D corner should project under the current camera estimate and where it was actually detected in the image, summed over all corners and views; this is a tiny bundle adjustment (lesson 05). Once calibrated, you can undistort images so epipolar geometry and triangulation behave.

标定就是恢复 K 和畸变系数的流程。标准方法是从多个角度给相机看一个已知的图案（棋盘格）：每个视角给出一批已知的三维角点位置及其被检测到的像素配对，你去求解那组内参、每个视角的外参以及畸变，使角点的重投影最吻合——即最小化重投影误差，也就是一个已知三维角点在当前相机估计下本应投影到的位置与它在图像中实际被检测到的位置之间的像素差距，对所有角点和所有视角求和；这是一个微型的光束平差（第 05 课）。一旦标定好，你就能对图像做去畸变，让对极几何和三角化表现正常。

Where this points next

这将指向何处

We now have the forward map: given a 3D point and a calibrated camera, we know exactly which pixel lights up — and we've proved that map is many-to-one in depth, so a single image cannot be inverted. Lesson 03 gave us correspondences; this lesson gave us the projection that produced them. Lesson 05 closes the loop by inverting the map with a second view: two rays from two known cameras intersect at one 3D point (triangulation), the essential/fundamental matrices encode the relationship between the two views' correspondences (K is exactly what converts one into the other), and for a rectified stereo pair the disparity-to-depth formula z = f·B/disparity turns pixel offsets directly into metres. The scale ambiguity we noted here — depth divides, depth destroyed — reappears there as the monocular scale problem that two calibrated views finally resolve. (From there, the companion 3D Vision track takes the geometry as its own discipline — rigid motion on SE(3), depth sensors and point clouds, and differentiable rendering.)

现在我们有了前向映射：给定一个三维点和一台已标定的相机，我们精确地知道哪个像素会亮——而且我们已经证明这个映射在深度上是多对一的，所以单幅图像无法被反演。第 03 课给了我们对应关系；本课给了我们产生这些对应的那次投影。第 05 课用第二个视角反演这个映射，从而闭合回路：来自两台已知相机的两条射线相交于一个三维点（三角化），本质/基础矩阵编码了两个视角对应关系之间的关系（K 恰恰是把其中一个转换成另一个的东西），而对一对校正过的立体图像，视差到深度的公式 z = f·B/disparity 直接把像素偏移变成米。我们在这里注意到的尺度模糊——深度做除法、深度被摧毁——在那里以单目尺度问题的形式再度出现，最终由两个已标定的视角来解决。（从那里往后，配套的 3D Vision 课程把几何当作一门独立的学科来处理——SE(3) 上的刚体运动、深度传感器与点云，以及可微渲染。）

Interview prompts

面试题

• What information is lost when a 3D point projects to a 2D image? (Depth: the whole ray collapses to one pixel because t cancels in f·tX/tZ.)当一个三维点投影到二维图像时，丢失了什么信息？（深度：整条射线坍缩到一个像素，因为 t 在 f·tX/tZ 中被约掉了。）
• What are camera intrinsics and extrinsics, and why split them? (K = camera, fixed; [R|t] = pose, per-frame.)什么是相机内参和外参，为什么要把它们拆开？（K = 相机，固定不变；[R|t] = 位姿，逐帧变化。）
• You crop by (a,b) and resize by (α,β) — how does K change? (Crop: c_x−a, c_y−b. Resize: scale f_x,c_x by α, f_y,c_y by β.)你按 (a,b) 裁剪、按 (α,β) 缩放——K 如何变化？（裁剪：c_x−a, c_y−b。缩放：把 f_x,c_x 乘以 α、f_y,c_y 乘以 β。）
• What does the homogeneous scale s represent? (The depth Z; dividing by it is the perspective divide.)齐次尺度 s 代表什么？（深度 Z；除以它就是透视除法。）
• Relate focal length to field of view. (FOV = 2·arctan(W/2f); longer f = narrower FOV.)把焦距与视场角联系起来。（FOV = 2·arctan(W/2f)；f 越长，FOV 越窄。）
• Why does radial distortion matter for measurement tasks? (It bends straight lines and breaks the epipolar assumption; must undistort before triangulating.)为什么径向畸变对测量类任务很重要？（它把直线弯曲、破坏对极假设；三角化之前必须先去畸变。）