The 3D problem and the representation zoo

The CV track's geometry arc ended at lesson 05 by surveying NeRF and Gaussian Splatting — naming them without deriving them. This track earns them. But before any method, one decision governs everything downstream: what object are we actually optimizing? A point cloud, a mesh, a grid of occupancy, a signed-distance function, a field of colored fog, a million fuzzy blobs? Each renders differently, optimizes differently, and fails differently. This lesson builds the loop that every later lesson is a move inside, and lays out the representation design space so the acronyms stop looking like a zoo.

1 · The inverse problem, and the loop that solves it

Start from the CV track's central loss. A pinhole camera maps a 3D point (X,Y,Z) to a pixel by x = f·X/Z, y = f·Y/Z (lesson 04). The Z in the denominator is the whole story: scale the point along its ray to (tX, tY, tZ) and the t cancels — f·tX/tZ = f·X/Z — so every point on a ray lands on the same pixel. Projection is many-to-one. An image is a function of the scene, and that function threw depth away.

Write the forward direction honestly. A scene S (geometry + appearance) and a camera produce an image by a rendering operator R:

3D vision is the inverse problem: recover S given one or more images I. Because R is many-to-one, the inverse is ill-posed — many scenes explain the same image (a small near object and a large far one are indistinguishable; the CV track called this the monocular scale ambiguity). You cannot simply invert R. What you can do is search the space of scenes for one whose renderings match the observations. That search is the loop this entire track lives inside:

Read the four steps as the table of contents for the track. Step ① is this lesson: which representation. Step ② is rendering — rasterization (lessons 06–07) or ray-marching (lesson 09). Step ③ is the loss — photometric for radiance fields, geometric (Chamfer, reprojection) for point clouds and depth. Step ④ is the inversion — and it only works if R is differentiable, the property we spend section 4 on. Classical geometry (CV 04–05: triangulation, SfM, ICP) is a special, hand-derived way of doing this loop for point representations; the modern era is about making step ② differentiable so a generic optimizer can do step ④.

2 · The representation zoo

There are only a handful of ways to write down 3D structure, and they split into two families by one question: do you store the shape, or a function that tests the shape?

Explicit representations store the geometry directly, as primitives with coordinates you can list:

• Point cloud — an unordered set of 3D points {(x,y,z)}, optionally with color/normal. The rawest output of every depth sensor (lesson 03). No connectivity, no surface, no "inside."
• Voxel grid — a 3D array; each cell holds occupancy, color, or a value. The direct 3D analog of a pixel image, which is exactly why it's tempting — and, as section 3 shows, exactly why it's expensive.
• Mesh — vertices joined into faces (triangles). The representation graphics hardware was built to draw. Compact for surfaces, but the connectivity is rigid: changing topology means re-meshing.

Implicit representations store a function f:ℝ³→ℝ and define the shape as a level set of it — the surface is "where f crosses a threshold." You never list the surface; you query the function.

• Occupancy field — f(x,y,z) = P(\text{inside}), surface at f = 0.5.
• Signed distance function (SDF) — f(x,y,z) = signed distance to the surface (negative inside), surface at f = 0. Carries more information than occupancy: not just whether you're inside but how far from the surface — which is what lets you march rays efficiently and compute normals as ∇f (lesson 05).
• Radiance field — the NeRF representation: f(x,y,z,\text{dir}) → (\text{color}, \text{density}). Not a hard surface at all but a continuous volume of colored, semi-transparent fog (lesson 09).

And the hybrid the field converged on for real-time work:

• 3D Gaussians — millions of little anisotropic blobs, each an explicit primitive (position, shape, color, opacity) that you rasterize like a mesh, yet soft and differentiable like a field. It keeps the optimizer-friendliness of implicit methods and the render-speed of explicit ones (lesson 10).

The whole zoo, on the axes that actually decide a design:

The widget below makes the explicit/implicit split tangible in 2D, where a "voxel grid" is a pixel grid and a "surface" is a curve. Flip between the four representations of the same shape and watch what each one stores — and what happens to each when you change the shape's topology.

// signed distance to a union of two disks: min of the two disk SDFs
function sdf(px, py){
  const dA = Math.hypot(px - ax, py - ay) - r;   // <0 inside disk A
  const dB = Math.hypot(px - bx, py - by) - r;
  return Math.min(dA, dB);                        // union = min (valid SDF outside)
}
const inside = (px,py) => sdf(px,py) < 0;

// VOXEL GRID: fill every cell whose centre is inside → cost is N×N cells (N×N×N in 3D)
for (let i=0;i<N;i++) for (let j=0;j<N;j++)
  if (inside(cellCx(i), cellCy(j))) fill(i,j);    // stored ∝ N²  →  the curse of dimension

// IMPLICIT FIELD: store nothing but the function; the "surface" is the level set {sdf=0}
//   render by colouring each pixel by sdf value and drawing the zero contour.

// POINT CLOUD / MESH: sample the BOUNDARY only (∝ its length). A point on disk A is on
//   the union boundary iff it is outside disk B:
for (const t of angles) {
  const p = onCircle(A, t);
  if (dist(p, B) > r) boundaryPts.push(p);        // stored ∝ perimeter, not area
}
// mesh = connect consecutive boundary points into segments (edges). Merging the disks
//   changes which points are on the boundary → the mesh must be re-connected; the grid,
//   field, and point cloud just... change, with no bookkeeping.

3 · The curse of the third dimension

Why not just use a voxel grid? It is the honest 3D analog of an image — a pixel image is a 2D grid of samples, so a 3D grid of samples should be the natural next step. The problem is arithmetic. An image at resolution N per side has N² pixels; a volume has N³ voxels. Resolution you take for granted in 2D becomes ruinous in 3D:

A 1024³ dense grid of floats does not fit in GPU memory, and a 3D convolution over it costs O(N³) multiply-adds per channel — the reason a naïve 3D CNN (lesson 11) is a non-starter at high resolution. Two escapes, and both drive real architectures:

• Store the surface, not the volume. Shapes are hollow — the interesting content is a 2D surface embedded in 3D, so its size scales like O(N²), not O(N³). Meshes and point clouds pay only for the surface, which is why they dominate graphics and sensing. The widget's grid mode fills area (grows fast with N); its point/mesh modes trace only the boundary (grows slowly).
• Exploit sparsity. Even as a volume, almost every voxel is empty. Sparse voxel structures (octrees, hash grids) and sparse convolution (lesson 11) store and compute only where there is content, turning O(N³) back into "∝ occupied cells." Instant-NGP's hash grid (lesson 09) is exactly this trick applied to a radiance field.

Implicit representations sidestep the grid entirely: an MLP that maps (x,y,z)→\text{value} has a fixed memory cost (its weights) and is queried at continuous coordinates, so there is no resolution to blow up. You trade the storage explosion for a per-query network evaluation — cheap to store, expensive to render. That trade is the single most important axis in the whole zoo, and it recurs in every lesson from here.

4 · Differentiability — the property that closes the loop

Return to step ④ of the loop. To update the representation S from the image error, we need the gradient ∂L/∂S, and by the chain rule that means differentiating through the renderer: ∂L/∂S = (∂L/∂Î)(∂Î/∂S). If ∂Î/∂S doesn't exist or is zero almost everywhere, gradient descent is dead in the water. This is the requirement that shapes modern 3D more than any other.

Where it bites: the standard graphics rasterizer (built in full in lessons 06–07) is not differentiable. Deciding which triangle covers a pixel is a discrete argmax with a hard edge — nudge a vertex and the pixel either flips to a new triangle or doesn't; the derivative is zero in between and undefined at the jump. So you cannot naively backprop an image loss into mesh vertices. The field's responses to this discontinuity define its landmarks:

• Make the geometry soft. Replace hard coverage with a smooth falloff. A radiance field (lesson 09) integrates continuous density along a ray, so a small change in the field makes a small, differentiable change in the pixel — this is precisely why NeRF trains by plain gradient descent on a photometric loss. Soft rasterizers and Gaussian Splatting (lesson 10) apply the same idea to explicit primitives: a Gaussian's soft edge gives every pixel a nonzero gradient w.r.t. the blob's position and shape.
• Keep an implicit function differentiable by construction. An MLP occupancy/SDF is smooth in its inputs and weights, so gradients flow freely — one reason learned shape models (lesson 05) went implicit.

Where this points next

We now have the frame: 3D vision is inverse rendering, run as a represent→render→compare→backprop loop; the representation you pick (explicit vs implicit) sets the memory cost, the rendering method, and whether the loop is even differentiable. But there is a prerequisite we quietly assumed all through step ①: to place a camera, align a scan, or move an object, we need to transform things in 3D — translate and, worse, rotate them. Translation is easy; rotation is not a vector space, and naïvely optimizing rotations breaks in ways that sink real systems. Lesson 02 builds rigid-body motion — SO(3), SE(3), quaternions, and the Lie-algebra trick that lets you do gradient descent on rotations without ever leaving the space of valid ones. Every later lesson that places a camera (registration, NeRF poses, Gaussian covariances, bundle adjustment) stands on it.

Interview prompts

• Why is single-image 3D reconstruction ill-posed? (§1 — projection divides by depth and is many-to-one; infinitely many scenes render to the same image, so the inverse needs a prior or extra views.)
• Contrast explicit and implicit 3D representations. (§2 — explicit stores geometry (fast render, hard to optimize, topology issues); implicit stores a function/level set (easy to optimize, any topology, slow to render).)
• Why not represent everything as a dense voxel grid? (§3 — O(N³) memory and compute; a 1024³ float grid is >4 GB. Store surfaces (O(N²)) or use sparsity.)
• Why can't you directly backprop an image loss into mesh vertices through a standard rasterizer? (§4 — coverage is a discrete argmax with zero/undefined gradient at edges; you need a soft/differentiable renderer.)
• Given a product — AR try-on, a self-driving perception stack, a text-to-3D tool — which representation would you reach for and why? (§2–§3 — argue from render speed, memory, editability, and the sensor/supervision available, not from novelty.)
• Where does classical SfM/SLAM sit in the inverse-rendering loop? (§1 — it's a hand-derived instance of the loop for point representations, with reprojection error as the loss and bundle adjustment as step ④.)