Generative 3D, dynamics, and 3D systems

Every lesson so far — from the depth sensors of lesson 03 through the radiance fields of lesson 09 and the Gaussians of lesson 10 to the learned depth of lesson 12 — recovered 3D from observations: something out there was measured, and we inverted the measurement. This capstone flips the arrow. Now there is no captured scene: we want to generate 3D from a prompt or a single image, to handle scenes that move (time — the fourth dimension), and to ship the result under real latency, memory, and sensor constraints. Then we name the metrics that decide whether any of it worked — and expose the ones that quietly lie.

1 · The obstacle for generation: there is no ImageNet-scale 3D

Recovery and generation look symmetric but are not, and the asymmetry is a data problem. To recover 3D you point a sensor at the world and it hands you geometry (lesson 03), or you photograph a scene from many angles and invert the renderer (lessons 09–10). To generate 3D you need a model that has seen enough 3D to know what a plausible chair, dragon, or street looks like from every angle. And that is exactly what does not exist at scale.

Count the orders of magnitude. Image-text pretraining runs on billions of image-caption pairs scraped from the web (LAION-scale); the 2D generative models of CV lesson 17 are trained on that firehose. The largest curated 3D asset collections — Objaverse and its successors — are on the order of a million to ten million objects, and most are untextured, inconsistently scaled, or low quality. 3D assets are expensive: each is authored by an artist or reconstructed by a capture rig, whereas an image is a single button-press. So the field faces a hard fact: you cannot train a 3D generator the way you train an image generator, because the 3D data is roughly three orders of magnitude scarcer.

2 · Route 1 — lift a 2D prior into 3D (Score Distillation / DreamFusion)

The first route is the cleverest, because it needs zero 3D data. The idea: a pretrained text-to-image diffusion model (CV lesson 17) already encodes, implicitly, what objects look like from every viewpoint — it has seen millions of chairs from every angle. So instead of training a 3D generator, distill that 2D knowledge into a single 3D asset by optimization.

Set up the loop exactly as in lesson 01 §1, but with a generative critic in place of a captured image. Let the thing we optimize be a 3D representation with parameters θ — a NeRF (lesson 09) or a set of 3D Gaussians (lesson 10). Repeat:

• Sample a random camera pose around the object.
• Render the representation from that pose through the differentiable renderer (lesson 01 §4) to get an image x = g(θ).
• Ask a frozen pretrained diffusion model: "how would you nudge this image to make it a more plausible sample for the text prompt y?" That nudge is the score — the diffusion model's estimate of the direction of increasing likelihood — and it is available for free from the network's noise prediction.
• Backpropagate that image-space nudge through the differentiable renderer into the 3D parameters θ.

This is Score Distillation Sampling (SDS), the engine of DreamFusion. Concretely, noise the render, let the diffusion U-Net \hat{ε}_\phi predict the noise, and the SDS gradient of the loss L w.r.t. the 3D parameters is (dropping a per-step weight w(t)):

Read it piece by piece. The bracket (\hat{ε}_φ − ε) is "predicted noise minus the noise we actually added" — the direction in image space that raises the render's likelihood under the diffusion prior for prompt y. The factor ∂x/∂θ is the Jacobian of the renderer: it carries that image-space push back into 3D parameters. The key trick that makes SDS cheap is that it skips differentiating through the diffusion U-Net's own Jacobian (which would be enormous) — it uses the noise-residual directly as the gradient. No 3D data appears anywhere: the loop distills a 2D model into one 3D asset.

Fixes, and why they work. Two lines of attack, each removing one root cause:

• Multiview-aware diffusion (MVDream). Fine-tune the 2D diffusion model to generate several consistent views at once, so the prior itself becomes 3D-aware and no longer wants every angle to be a front. This directly kills Janus by giving the critic viewpoint awareness.
• Variational SDS (ProlificDreamer). Replace the single-point-estimate SDS gradient with a variational objective (VSD) that models a distribution over renders rather than pushing toward one mode. This lifts the saturation and restores diversity, and allows much lower guidance.

The whole route is remarkable precisely because of the data argument in §1: it manufactures a 3D asset from a model that never saw 3D. Its cost is the flip side — it is a per-asset optimization, minutes to hours each, exactly the expense that Route 3 will amortize away.

3 · Route 2 — native 3D generation

The second route confronts the data gap head-on: train a generative model directly on 3D data (Objaverse and friends). The generative machinery is the same as in 2D — diffusion or autoregression — only the data type changes. The question becomes what 3D representation do you generate?, and the answer recapitulates the representation zoo of lesson 01:

• Point clouds — diffuse or autoregress a set of points (e.g. Point-E). Cheap, but no surface.
• Voxels — generate an occupancy/feature grid; simple but pays the O(N³) tax of lesson 01 §3.
• Meshes — autoregress vertices and faces directly (e.g. token-by-token mesh generation) for artist-ready output.
• Latent codes — the dominant modern choice: encode shapes into a compact latent (a VAE/autoencoder) and run diffusion in that latent space, mirroring the latent-diffusion trick of CV lesson 17.

The representation that made 3D-aware generation practical is the triplane (introduced by EG3D). The obstacle is exactly lesson 01 §3: a dense 3D feature volume is O(N³) and blows up memory. The triplane factors that volume into three orthogonal 2D feature planes — one each in the XY, XZ, and YZ planes. To read the feature at any 3D point (x,y,z), project it onto all three planes, bilinearly sample each, sum (or concatenate) the three feature vectors, and decode with a tiny MLP into color and density:

Three wins fall out at once, and they are why the triplane is the backbone of so many 3D-aware GANs and diffusion models. Memory drops from O(N³) to O(3N²) — surface-scale, not volume-scale. The planes are ordinary 2D feature maps, so you can generate them with a standard 2D CNN or 2D diffusion backbone, reusing all the mature 2D generative machinery. And the tiny decoder MLP keeps the whole thing a continuous field you can volume-render (lesson 09) and differentiate. The triplane is, in effect, the compromise the field found between the storage explosion of voxels and the query cost of a pure MLP.

4 · Route 3 — feed-forward reconstruction (LRM)

Routes 1 and 2 still, in the image→3D setting, tend to optimize per scene. Route 3 removes that entirely. A Large Reconstruction Model (LRM) is a transformer trained on large multiview/3D datasets that maps one or a few images directly to a 3D representation in a single forward pass — typically a triplane (as in §3) or a set of Gaussians (lesson 10) — in seconds, with no per-scene optimization.

This is the sharpest contrast in the whole track. Recall how lessons 09–10 produce a 3D asset: you take posed images of one scene and run gradient descent for minutes-to-hours on that scene's parameters, from scratch, every time. NeRF and 3DGS are per-scene optimizations — brilliant fits, but the work is not reusable across scenes. LRM amortizes that optimization into a network: it pays the cost once, at training time, over many scenes, and thereafter reconstruction is a forward pass. Put side by side:

The trade is generalization for speed: an LRM cannot exceed what its training data taught it, whereas a per-scene NeRF will happily overfit any scene you give it. But for the image→3D problem — "here is a photo, give me the object in 3D, now" — feed-forward amortized reconstruction is the current sweet spot, and it is where the field's momentum sits.

5 · Dynamics — the fourth dimension (4D)

Everything up to here, generation included, assumed a static scene. Real scenes move: people walk, cloth flaps, cars drive. Adding time turns 3D into 4D, and the representations of lessons 09–10 extend in two natural ways.

Dynamic NeRF / 4D Gaussian Splatting. Two strategies dominate, and they map onto the implicit/explicit split:

• Deformation field (implicit-leaning). Keep a single canonical radiance field (the object at rest) and learn a time-dependent warp D(x, t) that maps each point at time t back into canonical space before querying: c,σ = F_canonical(x + D(x,t)). Motion is factored out from appearance, which is compact and interpolates smoothly.
• Per-primitive trajectories (explicit-leaning). With 3D Gaussians, give each blob its own motion trajectory over time (its center, and often rotation/scale, as functions of t). The scene is then just the Gaussians of lesson 10, animated — which keeps 3DGS's real-time rendering while adding time.

Parametric models for the special case that matters most: humans. A general deformation field is unconstrained; but humans (bodies, faces, hands) are so common and so structured that the field built low-dimensional parametric mesh generators for them, fit from data:

Each is a differentiable function M(θ, β) → \text{mesh}: feed a few dozen numbers, get a full mesh. Crucially the rotations here are exactly the SO(3) joint rotations of lesson 02 — SMPL's pose θ is a stack of axis-angle rotations, optimized on the manifold. Because the parameter count is tiny, you fit the model to images or point clouds by optimizing those few parameters (the inverse-rendering loop of lesson 01, restricted to a low-dimensional prior). This is what powers markerless motion capture, avatar/AR try-on, and animation: the parametric model supplies the missing 3D prior that makes an otherwise ill-posed fit well-posed.

6 · 3D systems and deployment — the CV 19 analog

This section is to this track what CV lesson 19 was to the 2D track: the shift from "does the method work in a paper" to "does the system survive contact with a product." Three parts: representation by product, evaluation metrics and their pitfalls, and the silent failures nobody puts in the paper.

(a) Representation by product

The whole track's refrain — choose the representation from the constraint, not the fashion (lesson 01) — is decided here. Four products, almost no shared design choices:

Two of these deserve a note. Robotics leans on TSDF fusion (truncated signed distance function) — the KinectFusion recipe: integrate each depth frame into a running voxel SDF, averaging out sensor noise into a clean surface you can both render and query for collisions. And autonomous stacks increasingly predict occupancy directly (lesson 11) precisely because a bounding-box detector fails on the object classes it was never trained on, whereas "is this volume occupied?" degrades gracefully.

(b) Evaluation metrics — and their pitfalls

Each 3D task has its own metric; you must know what each measures and where each lies.

(c) The silent failures

These sink real 3D systems and appear in no benchmark:

• Calibration and extrinsics. Every multi-sensor rig assumes you know each sensor's intrinsics and its pose relative to the others (the SE(3) extrinsics of lesson 02). If a LiDAR-to-camera transform is off by a few centimeters or a fraction of a degree, projected points land on the wrong pixels and fusion silently corrupts. No model fixes a miscalibrated rig — the error is upstream of the network.
• Time synchronization. A moving car at 15 m/s travels 15 cm in 10 ms. If the camera and LiDAR are not hardware-time-synced, a fast-moving object is in different places in the two sensors' data, and fusion smears it. Sync is a systems problem, not a learning one.
• Annotation cost. Labeling 3D is far more expensive than 2D — annotators draw boxes in BEV/3D and the labels must be propagated across frames of a sequence. This is why 3D datasets are small (the §1 problem again, now on the supervised side).
• Sim-to-real gap. The tempting escape from annotation cost is synthetic data, but a model trained in simulation faces a domain gap — sensor noise, materials, and lighting differ — and often degrades on real data unless the gap is explicitly closed (domain randomization, real fine-tuning).

Where this leaves you — the whole track in one line

Step all the way back. Lesson 01 claimed the entire field is a single loop: pick a 3D representation → render it → compare to observations → backprop the error into the representation, and it closes only if the renderer is differentiable. Every one of the ten lessons was a move inside that loop. Lesson 02 gave us the SE(3) machinery to place things in it and optimize their poses on the manifold. Lessons 03–05 supplied the observations (depth, point clouds), aligned them (ICP), and turned them into surfaces (meshes, SDFs). Lessons 09–10 built the differentiable renderers — volume rendering / NeRF, then Gaussian Splatting — that let ordinary images close the loop, and 08–09 learned across scenes (3D backbones, detection) and pulled depth from single images.

This capstone did the one thing the loop had left implicit. Every prior lesson ran the loop with the observations in hand — the arrow pointed from a measured scene back to its 3D. Generation runs the same loop with the observation replaced by a prior: SDS swaps the captured image for a frozen 2D diffusion critic; native 3D and LRM move the optimization to training time so inference is a forward pass; 4D adds time to the representation; and the systems section decides which representation, rendered how, judged by which metric, a real product can afford. There is no new machine here — only the recognition that recovering 3D and generating 3D are the same inverse-rendering loop, differing only in whether the thing on the right-hand side is a photograph or a prior. That is the whole track, and it points back to where it started: the track index and its four questions — representation, rendering, recovery, system.

// nearest-neighbour distance from point p to set Q (brute force)
function nnDist(p, Q){
  var best = Infinity;
  for (var i=0;i<Q.length;i++){
    var dx=p.x-Q[i].x, dy=p.y-Q[i].y, d=dx*dx+dy*dy;
    if (d<best) best=d;
  }
  return Math.sqrt(best);
}
// directional Chamfer = MEAN of nearest-neighbour distances (one direction)
function chamferDir(A, B){
  var s=0; for (var i=0;i<A.length;i++) s+=nnDist(A[i], B); return s/A.length;
}
var cGP = chamferDir(GT, PRED);        // GT → pred
var cPG = chamferDir(PRED, GT);        // pred → GT
var symmetric = 0.5*(cGP + cPG);       // symmetric Chamfer (average form)

// F-score@τ = harmonic mean of precision and recall of points within τ
function frac(A, B, tau){               // fraction of A within tau of some B point
  var n=0; for (var i=0;i<A.length;i++) if (nnDist(A[i],B)<=tau) n++;
  return n/A.length;
}
var precision = frac(PRED, GT, tau);    // preds that are near a GT point
var recall    = frac(GT, PRED, tau);    // GT points that are covered by a pred
var F = (precision+recall>0) ? 2*precision*recall/(precision+recall) : 0;

// KEY: adding one far outlier to PRED spikes chamferDir(PRED,GT) (its NN dist is huge,
//   and it enters the MEAN) while precision drops by only 1/|PRED| → F-score barely moves.

Interview prompts

• Why can't you train a text-to-3D model the way Stable Diffusion was trained, and what are the three ways around it? (§1 — no web-scale 3D data (~1000× scarcer than images); lift a 2D prior (SDS), native 3D generation, or feed-forward reconstruction.)
• Explain Score Distillation Sampling and the Janus artifact. (§2 — optimize a NeRF/3DGS so random-view renders please a frozen 2D diffusion prior, backprop through the differentiable renderer; per-view supervision biased to canonical fronts yields repeated faces — fix with multiview diffusion / variational SDS.)
• What is a triplane and what problem does it solve? (§3 — factor an O(N³) feature volume into three 2D planes + a tiny MLP; cuts memory to O(N²) and lets 2D CNN/diffusion backbones generate 3D-aware content.)
• Contrast an LRM with per-scene NeRF/3DGS optimization. (§4 — NeRF/3DGS optimize this scene's parameters (minutes–hours); LRM amortizes reconstruction into a network trained over many scenes, giving image→3D in one forward pass, bounded by its training distribution.)
• How do you add time to a NeRF or Gaussian scene, and how do you make an ill-posed human fit well-posed? (§5 — a deformation field warping to a canonical space, or per-Gaussian trajectories; fit a low-dimensional parametric model (SMPL/FLAME/MANO) whose few pose θ/shape β parameters supply the missing prior.)
• Your reconstruction has a great Chamfer distance but looks wrong — what happened, and what would you report instead? (§6 — Chamfer is dominated by outliers and rewards coverage over detail; report F-score@τ (robust) and, for novel views, LPIPS alongside PSNR since PSNR ≠ perceptual quality.)