3D Vision, from first principles

A linearized continuation of the Computer Vision track into three dimensions. That track ended its geometry arc (lesson 05) by surveying NeRF and 3D Gaussian Splatting; this track derives them — and everything around them — from the ground up. The goal is the same: coverage without memorization. Know which representation holds the 3D, how it renders back to the pixels we can measure, how we invert that to recover it, and which system constraint decides the whole design.

Who this is for

You finished the 2D CV track (or know cameras, projection, and stereo already) and now face the 3D half of the field: point clouds, meshes, SDFs, NeRF, Gaussian splatting, LiDAR detection, monocular depth, and text-to-3D. In interviews these feel like a pile of unrelated systems. This track puts them on one line, so each lesson answers why the next one has to exist.

The first-principles frame

A 2D image is a measurement of a 3D world, and lesson 04 of the CV track proved the measurement is lossy in one specific, fatal way: projection is many-to-one along the ray, so depth is destroyed. All of 3D vision is the fight to get that depth back — and then to represent, render, learn on, and generate the structure we recover. Strip away the acronyms and every method is a move in one loop:

3D representation ──render (forward)──▶ predicted 2D ──compare──▶ observed 2D points / mesh / │ rasterize or loss (images, voxels / SDF / │ ray-march │ depth, scans) radiance field / │ │ Gaussians ◀─────────── backprop ─────────┘ (invert: update the 3D)

Fix a representation, define how it renders to the sensor, then optimize the representation until its renderings match what we observed. The catch that organizes the entire modern field: to invert, the renderer must be differentiable. That single requirement is why NeRF and Gaussian Splatting exist, and why so much of the track is about rendering equations. Everything reduces to four questions.

Representation

Points, voxels, mesh, occupancy, SDF, radiance field, Gaussians

Rendering

Rasterize vs ray-march; and can we backprop through it?

Recovery

Sensors, triangulation, registration, learned depth, fitting

System

Which representation for which product; latency, memory, sensors, metrics

The tension that runs through everything

Two families of representation, and the entire track is a dialogue between them:

	Explicit	Implicit
Examples	Point cloud, mesh, voxels, 3D Gaussians	Occupancy field, SDF, radiance field (NeRF)
Stored as	Primitives with explicit coordinates	A function f(x,y,z) — usually a network
Rendering	Fast — rasterize the primitives	Slow — march rays and query the field
Optimizing	Hard — fixed count, rigid topology	Easy — continuous, any topology, differentiable
Editing / memory	Easy to edit; memory ∝ detail	Hard to edit; memory = network size

Meshes and points are easy to draw but painful to optimize and can't change topology. Implicit fields optimize beautifully and handle any shape but are expensive to render. Almost every method you'll meet is either picking a side for a product constraint or bridging the two (marching cubes pulls a mesh out of an SDF; Gaussian Splatting keeps the differentiability of a field but renders explicit blobs in real time). Keep this table in your head and the field stops looking like a zoo.

Why this order

We start with representations and rigid motion, because you cannot recover or render anything you cannot first store and place. Then we get real data (depth sensors, point clouds), align it (registration), and turn it into surfaces (meshes, SDFs). Then we render — first the classical rasterization pipeline that draws a mesh to pixels in real time, whose hard, non-differentiable coverage test is exactly what motivates the differentiable renderers (NeRF, Gaussian Splatting) that let images alone supervise 3D. With rendering in hand we learn across scenes (3D deep nets, detection), recover depth from ordinary images (monocular/self-supervised/MVS), and finally generate 3D and ship it under real constraints.

Stage	Lessons	What you should be able to do
Foundations	01–02	Choose a representation from its trade-offs; compose and optimize rigid transforms without leaving SO(3).
Data & geometry	03–05	Turn sensors into point clouds, align scans with ICP, and extract surfaces via meshes and SDFs.
Forward rendering	06–08	Rasterize a mesh to pixels — projection, coverage, the z-buffer, texturing — then climb to photorealism (PBR, shadows, global illumination), and see why it's still non-differentiable.
Differentiable rendering	09–10	Derive volume rendering and NeRF, then Gaussian Splatting, and explain the real-time gap.
Learning	11–12	Build permutation-invariant 3D backbones, detect in 3D, and recover depth from images.
Frontier & systems	13	Generate 3D, handle dynamics, and pick a representation + metric for a real product.

The lessons

The 3D problem and the representation zoo

The inverse-rendering loop, why depth is lost, and the design space of 3D representations: point cloud, voxel, mesh, occupancy, SDF, radiance field, Gaussians — explicit vs implicit, memory scaling, and differentiable-render-ability.

Rotations and rigid-body motion — SE(3)

Why orientation isn't a vector space: rotation matrices, Euler angles and gimbal lock, axis-angle, quaternions, the so(3)/se(3) Lie algebra, exp/log (Rodrigues), and why bundle adjustment and ICP optimize on the tangent space.

Depth and point clouds

How 3D data is acquired — stereo, structured light, time-of-flight, LiDAR — plus back-projection with K⁻¹, metric vs inverse depth, and point-cloud operations: kNN, normals, voxel downsampling, outlier removal.

Registration and ICP

Aligning scans into one frame: the closed-form Procrustes/Kabsch solution with the reflection fix, iterative closest point (point-to-point vs point-to-plane), local minima and initialization, global registration, and the bridge to pose graphs.

Surfaces — meshes and implicit fields

From points to surfaces two ways: explicit triangle meshes (normals, Poisson reconstruction) and implicit level sets (occupancy, signed distance functions, the eikonal property, DeepSDF), with marching cubes as the bridge between them.

The rasterization pipeline

The forward renderer the whole field runs on: model→view→clip→NDC→screen transforms and the perspective divide, triangle coverage via edge functions and barycentric coordinates, back-face culling and clipping, and the z-buffer for hidden-surface removal — how a mesh becomes pixels in real time.

Shading, interpolation, and textures

Making the pixels look right: perspective-correct attribute interpolation, texture mapping with mipmapping (the CV aliasing problem again), flat/Gouraud/Phong shading and the programmable shader pipeline, MSAA — and why the hard coverage test makes rasterization non-differentiable, forcing the soft renderers that follow.

Realistic rasterization — PBR, shadows & global illumination

Climbing from local shading to photorealism, all on the rasterizer: the rendering equation, physically-based materials (the microfacet/GGX BRDF, metallic-roughness, Fresnel), shadow mapping, global-illumination fakes (SSAO, IBL, lightmaps, hybrid ray tracing), and linear/HDR/tone-mapping — then why even photoreal forward rendering stays non-differentiable and hand-authored.

Volumetric rendering and NeRF

The volume rendering integral derived from light through participating media, discretized into alpha compositing; positional encoding and spectral bias, view dependence, hierarchical sampling, the photometric training loop, and NeRF's failure modes.

3D Gaussian Splatting

The 3D Gaussian primitive (covariance as scale×rotation), EWA projection to 2D, differentiable tile-based rasterization, the same front-to-back compositing as NeRF, spherical-harmonic color, adaptive density control, and why it hits 100+ FPS.

Deep learning on 3D and scene understanding

Why grid CNNs fail on point clouds: PointNet permutation invariance, PointNet++ hierarchy, sparse/voxel convolution, and the task heads — 3D detection (PointPillars, CenterPoint, BEV), segmentation, and occupancy prediction.

Learned depth — monocular, self-supervised, MVS

Depth as a prior problem: supervised and relative-vs-metric monocular depth (MiDaS, Depth Anything), self-supervised depth from video via photometric reprojection, and learned multi-view stereo with plane-sweep cost volumes.

Generative 3D, dynamics, and 3D systems

Creating 3D with no 3D data: Score Distillation (DreamFusion), triplanes, and feed-forward reconstruction (LRM); dynamic/4D scenes and SMPL body models; then evaluation (Chamfer, F-score, LPIPS), sensors, and picking a representation per product.

How to use this

Read it linearly once. Gaussian Splatting reuses NeRF's compositing equation and lesson 02's quaternions. NeRF reuses the alpha-blend the CV survey mentioned. ICP reuses SE(3). Registration reuses normals from the point-cloud lesson. The dependencies are real; skipping breaks them.
Answer in the four-question frame. "Use a NeRF" is weak. "This is a static scene, offline, where I need photorealistic novel views and don't care about seconds-per-frame, so an implicit radiance field beats a mesh" is the right altitude.
Name the missing constraint. Most 3D problems are ill-posed. Monocular depth needs a scale anchor; SDS needs a 2D prior; registration needs an initialization; NeRF needs posed views. The interview answer is usually which constraint is missing and how you supply it.
Tie the model to the sensor and the product. An AR headset, a self-driving stack, a content-creation tool, and a warehouse robot all do "3D" and share almost no design choices. Latency, memory, sensor suite, and metric decide everything.

Companion material

This track assumes CV lessons 04 (cameras & projection) and 05 (multi-view, stereo & SLAM) — the pinhole model, intrinsics K, extrinsics (R,t), and epipolar geometry. Use CV 17 (generative vision) for the diffusion background that lesson 12 lifts into 3D, and gpu_kernels when the interview turns from geometry to the rasterizer's throughput.