Deep learning on 3D and scene understanding

Lessons 09–10 fit one scene by optimization — a NeRF or a cloud of Gaussians is tuned until its renderings match a specific set of photos. That machinery reconstructs beautifully, but it does not recognize anything: point a NeRF at a new street and it has learned nothing transferable, no notion of "car" or "pedestrian," no ability to generalize to a scene it never optimized. To learn across scenes — to build a detector or a segmenter — we need networks that consume 3D directly. But the raw 3D from lesson 03, the point cloud, is an unordered, irregular, sparse set, and the grid convolution that powers image networks (CV lesson 06) simply does not apply to it. This lesson builds the 3D backbones that fix that, and the task heads that turn features into boxes, labels, and occupancy.

1 · Why grid CNNs fail on 3D, and the three families

The 2D convolution of CV lesson 06 rests on two properties of an image that we never had to think about. First, the pixels sit on a regular grid: every pixel has neighbors at fixed offsets (±1, 0), (0, ±1), and a 3×3 kernel is just "multiply these nine specific neighbors by these nine specific weights." Second, that grid gives a canonical ordering: pixel (i,j) is a well-defined slot, so weight-sharing across positions is meaningful. Translation equivariance — the whole reason CNNs work — is a statement about sliding a kernel across that lattice.

A point cloud (lesson 03) has neither property. It is a set: { p₁, …, p_n } with p_i ∈ ℝ³, and three facts about it break the CNN outright:

• Unordered. The cloud is a set, so any permutation of the n points is the same cloud. A function that consumes it must give the same answer regardless of the order they arrive in — there is no "point 1" the way there is a "pixel (0,0)." A CNN's very first assumption (fixed slots) is violated.
• Irregular / varying density. Points are scattered at continuous coordinates, not snapped to a lattice. Near a LiDAR sensor they are dense; far away they thin out. There are no fixed neighbor offsets to convolve over.
• Sparse. The points occupy a tiny fraction of the 3D volume they live in — the surfaces, not the empty air between them (exactly lesson 01's observation that shapes are hollow).

There are, broadly, only three ways to get a deep network onto this data, and every 3D architecture is a variation on one of them:

Read those as three answers to one question — how do I recover a grid and an ordering, or avoid needing them? Family A avoids the grid entirely by engineering invariance into the architecture. Family B rebuilds the grid but refuses to pay for the empty parts. Family C throws away one dimension to reuse the mature 2D machinery wholesale. The rest of the lesson builds each, in that order, then hangs task heads off them.

2 · Point-based — PointNet and PointNet++

Start with Family A, because its core idea is the cleanest and most quoted result in 3D deep learning. We want a function f that eats a set of points and returns a single global feature vector (a descriptor of the whole shape), and the one hard requirement is permutation invariance: reorder the input points and the output must not change, because a set has no order to begin with.

The PointNet trick. How do you build a function of a set that ignores order? Process each point independently with the same function, then combine the results with an operation that is itself order-blind. Concretely, apply a shared MLP h to every point (the same weights for all of them — this is where permutation symmetry enters, because "the same function on each element" cannot depend on which element came first), then aggregate the per-point features with a symmetric function — PointNet uses element-wise max-pooling:

Read the pieces. h lifts each 3-vector to a high-dimensional per-point feature (say 1024-D) with a small MLP shared across all points. The element-wise \operatorname{MAX} over the n points collapses them into one 1024-D vector: for each of the 1024 feature channels, keep the largest value across all points. Because \max\{a, b, c\} = \max\{c, a, b\} — max does not care about order — the pooled vector is provably invariant to any permutation of the inputs. Finally γ is another MLP mapping that global descriptor to a class label (or whatever the task needs). Sum or average would work as symmetric functions too; max is used because it tends to latch onto the most salient points and empirically generalizes best.

A beautiful consequence: for each channel, exactly one input point supplied the winning value. Those winners are the critical points — a sparse subset that alone determines the descriptor. Perturb any other point and the output is unchanged; PointNet has, in effect, learned a data-dependent skeleton of the shape. (The widget below makes both the invariance and the critical points visible.)

The T-Net. One more piece for robustness to pose: PointNet prepends a small learned network (the T-Net) that regresses an alignment transform — a 3×3 matrix — and applies it to the input points before the shared MLP, so that clouds arriving in slightly different orientations are canonicalized first. (It is a learned, data-driven cousin of the rigid alignment from lesson 02, though PointNet does not constrain it to SO(3).)

PointNet++ — go hierarchical. The fix is to reintroduce locality the way a CNN does — by building a hierarchy of growing receptive fields — but adapted to an unordered, irregular set. One set-abstraction layer does three things:

1. Sample a subset of centroids with farthest point sampling (FPS): greedily pick the point farthest from all already-picked points, repeating until you have the desired count. FPS spreads centroids evenly over the surface regardless of density — far better coverage than random sampling, which over-picks dense regions.
2. Group each centroid's local neighborhood by a ball query (all points within radius r) or kNN, producing a small local patch per centroid.
3. Encode each patch with a mini-PointNet (shared MLP + max-pool over just that patch) into one feature vector attached to the centroid.

The output is a smaller set of points, each now carrying a feature that summarizes its local region. Stack these layers and the receptive field grows: local edges → parts → whole object — precisely the local-to-global feature hierarchy of a CNN (CV lesson 06), just expressed as repeated sample→group→pool on a set instead of stride→convolve→pool on a grid. Multi-scale grouping (several radii per centroid) further hardens it against the varying density that plagues LiDAR.

3 · Voxel-based and sparse convolution

Family B takes the opposite tack: if the trouble is that a point cloud has no grid, give it one. Voxelize — partition space into a 3D lattice and mark each cell by the points that fall in it — and now an ordinary 3D convolution applies, the honest volumetric analog of the 2D CNN. This is conceptually the most direct port of image networks to 3D.

The catch is exactly the curse of the third dimension from lesson 01. A dense grid at resolution N per side has N³ voxels, and a 3D convolution costs O(N³) multiply-adds per channel. A 512³ grid is 134 million cells; a dense 3D CNN over it does not fit on a GPU. And it is almost all waste: the points sit on surfaces, so the vast majority of voxels are empty — you would be convolving over air.

Sparse convolution is the escape, and it is lesson 01's sparsity trick made into an operator. Store and compute only at occupied voxels. A hash map holds the active sites; the convolution gathers each active site's active neighbors, applies the kernel, and scatters the result — touching zero empty cells. Submanifold sparse convolution goes further and keeps the active set fixed (an output site is active only if the corresponding input site was), which prevents the "dilation" that would otherwise fill space with active voxels layer by layer and destroy the sparsity you were exploiting. Libraries like MinkowskiEngine and SpConv make this practical, and it is what makes high-resolution 3D CNNs tractable — the dominant backbone for LiDAR point clouds and dense indoor scans.

Projection / BEV — the fast shortcut. Family C sidesteps 3D convolution altogether. Flatten the cloud to a 2D image and run a mature, fast 2D CNN on it:

• Range image — unroll a spinning LiDAR's returns into a (\text{azimuth} × \text{elevation}) 2D grid, one "pixel" per beam direction, with depth as the value. It is the sensor's native layout, so it is dense and cheap.
• Bird's-eye view (BEV) — project points down onto the ground plane into a top-down grid. This is enormously convenient for driving because in BEV objects don't occlude each other, scale is roughly constant (no perspective foreshortening), and metric distances are preserved — so a 2D detector on a BEV grid reads out physical box positions directly.

Projection trades away some geometry (collapsing a dimension loses fine vertical structure and can merge distinct objects) for a large speed win. Point-voxel hybrids such as PVCNN keep the best of both: a point-based branch preserves fine detail at continuous coordinates while a low-resolution voxel branch does cheap neighborhood aggregation, fused each layer. A compact comparison:

4 · Task heads — the 3D analog of CV detection and segmentation

A backbone produces features; a head turns them into a prediction. The tasks mirror 2D vision (CV lesson 09) one dimension up, with the extra geometry making both the outputs and the pitfalls richer.

3D object detection. Instead of a 2D box (x, y, w, h), predict a 7-DOF oriented box: center (x, y, z), size (l, w, h), and heading \text{yaw} (rotation about the vertical axis — objects on the ground rarely pitch or roll, so one angle suffices). The families of section 3 each spawn a detector:

• VoxelNet / SECOND — voxelize, then a sparse-conv backbone + detection head. SECOND made the sparse convolution efficient enough to be the workhorse LiDAR detector.
• PointPillars — collapse each vertical column of points into a "pillar," encode it, and scatter into a 2D BEV pseudo-image so an ordinary 2D CNN does the detection. Skipping 3D convolution entirely makes it very fast, which is why it is widely deployed in autonomous vehicles.
• CenterPoint — anchor-free: instead of tiling oriented anchor boxes (which are awkward under rotation), detect each object's center as a peak in a BEV heatmap and regress the box attributes from that peak. Simpler and stronger, the 3D echo of anchor-free 2D detectors from CV lesson 09.

Camera-only 3D detection. Detecting 3D boxes from images alone (no LiDAR) is attractive — cameras are cheap — but ill-posed, because a single image lost depth (lesson 01's many-to-one projection). The modern recipes recover it:

• LSS (Lift-Splat-Shoot) — "lift" each pixel into a set of 3D points along its ray by predicting a distribution over depth, "splat" those features into a BEV grid, then detect in BEV.
• BEVFormer — use transformer queries anchored to BEV grid cells that attend into the multi-camera image features, learning the image→BEV mapping rather than fixing it via explicit depth.
• Pseudo-LiDAR — predict a depth map (lesson 12's territory), back-project it into a synthetic point cloud, and feed that to an off-the-shelf LiDAR detector. The striking finding was that how you represent the depth — as a point cloud rather than as extra image channels — mattered enormously.

Metrics. The 2D detection toolkit carries over with a dimension added: 3D IoU (volumetric intersection-over-union of two oriented boxes), 3D NMS to suppress duplicates, and mAP averaged over classes and thresholds. The nuScenes NDS (nuScenes Detection Score) additionally rolls in errors in translation, scale, orientation, velocity, and attribute — because in 3D a box can have the right IoU yet the wrong heading or size, which a pure IoU metric would let slide.

Segmentation. Assign a label per point rather than per box: semantic (road / building / vegetation), instance (this car vs. that car), or panoptic (both at once). Sparse-conv U-Nets dominate indoor and LiDAR semantic segmentation; KPConv defines a genuine convolution directly on points via learnable kernel points in continuous space (no voxel grid), a point-native answer to "what is a convolution on an irregular set?"

Occupancy prediction. A pointed current trend in autonomous driving: instead of emitting boxes, predict a dense voxel grid of occupancy + semantics for the whole scene — "is this cell occupied, and by what?" The motivation is a failure mode of boxes. A 7-DOF box assumes every obstacle is a known, roughly box-shaped category; but the road throws up open-set, oddly-shaped hazards — a fallen ladder, debris, an overhanging branch, a strangely articulated vehicle — that no box class fits and that a detector may miss entirely. An occupancy grid is category-agnostic: it can flag "something solid is here, do not drive into it" without ever naming it, which is exactly the robustness a safety-critical stack wants.

Widget — permutation invariance: why PointNet max-pools

Below are eight fixed 2D points, each labeled by its current position in the input list. For every point we compute a simple per-point feature h(p_i) = [\,x,\ y,\ x·x,\ \sin(3y)\,] — the stand-in for PointNet's shared MLP. From those we build two global descriptors: the PointNet one, an element-wise \operatorname{MAX} over all points (a symmetric aggregation), and a naive one that weights each point by its list index, Σ_i\, i·h(p_i) (order-dependent by design). Hit shuffle to permute the list order and relabel the points. Watch: the max-pooled vector is unchanged — it is a function of the set — while the index-weighted vector changes every time. The starred points are the critical points: for each feature channel, the one point that supplied the winning max.

// per-point feature: h(p) = [x, y, x*x, sin(3y)]  (x,y normalised to [-1,1])
function h(p){ return [p.x, p.y, p.x*p.x, Math.sin(3*p.y)]; }

// (1) PointNet-style: element-wise MAX over all points — a SYMMETRIC aggregation.
//     max{a,b,c} = max of the SET, independent of order → permutation-invariant.
var mx = [-Infinity,-Infinity,-Infinity,-Infinity], argmax = [0,0,0,0];
for (var i=0;i<pts.length;i++){
  var f = h(pts[i]);
  for (var d=0;d<4;d++) if (f[d] > mx[d]){ mx[d]=f[d]; argmax[d]=i; }  // track critical point
}

// (2) NAIVE order-dependent: weight each point by its LIST INDEX, then sum.
//     depends on the order the points arrive in → NOT a set function.
var naive = [0,0,0,0];
for (var j=0;j<pts.length;j++){
  var g = h(pts[j]);
  for (var d2=0;d2<4;d2++) naive[d2] += j * g[d2];   // the index j is the whole problem
}

// shuffle = permute the list order (Fisher–Yates). mx is UNCHANGED (max ignores order);
// naive CHANGES because the index weights land on different points.

Where this points next

We can now run networks on 3D — invariant to the unordered set (PointNet++), tractable on high-resolution volumes (sparse conv), fast on flattened views (BEV) — and read out boxes, labels, and occupancy that generalize across scenes. But look back at camera-only detection: every one of those methods (LSS, BEVFormer, pseudo-LiDAR) leaned on a depth it had to predict from images. We invoked it as a black box. Lesson 12 opens it: learned depth — how a network estimates a depth map from a single image (monocular, resolving the scale ambiguity with data priors) or from several (deep multi-view stereo, which learns the matching that classical MVS did by hand). That is the bridge from "3D from sensors" to "3D from ordinary pixels," and it feeds directly back into the pseudo-LiDAR and BEV detectors we just built.

Interview prompts

• Why can't you run a standard CNN on a raw point cloud? (§1 — a convolution needs a regular grid and a fixed neighbor ordering; a point cloud is an unordered, irregular, varying-density set with neither.)
• How does PointNet achieve permutation invariance, and why max-pool specifically? (§2 — a shared MLP h per point then a symmetric aggregation; \operatorname{MAX} ignores order so f=γ(\operatorname{MAX}_i\, h(p_i)) is invariant, and max latches onto salient "critical points.")
• What does PointNet miss, and how does PointNet++ fix it? (§2 — a single global pool captures no local neighborhood structure; PointNet++ builds a hierarchy: FPS-sample centroids, ball-query/kNN group, mini-PointNet per group, repeat — a growing receptive field like a CNN.)
• Why is a dense voxel 3D CNN usually infeasible, and what is the fix? (§3 — O(N³) memory/compute and mostly empty; sparse/submanifold convolution computes only at occupied voxels, restoring cost ∝ occupancy.)
• Contrast LiDAR and camera-only 3D detection. (§4 — LiDAR gives metric depth directly (PointPillars, CenterPoint on sparse conv/BEV); camera-only must recover depth — LSS lifts by predicted depth, BEVFormer uses BEV transformer queries, pseudo-LiDAR synthesizes a cloud — and is scored by 3D IoU/mAP and nuScenes NDS.)
• Why predict an occupancy grid instead of 3D boxes? (§4 — boxes assume known, box-shaped categories; a dense category-agnostic occupancy grid flags open-set, oddly-shaped obstacles a detector would miss, which safety-critical AV stacks want.)