Generative 3D, dynamics, and 3D systems

The track has built both directions: graphics maps a 3D scene to images, while reconstruction maps observations back to 3D. This capstone asks what happens when measurements are incomplete or absent and a learned prior must supply the missing structure: generate 3D from text or a single image, handle scenes that move (time — the fourth dimension), and ship the result under real latency, memory, sensor, and editability constraints. Then we name the metrics that decide whether any of it worked — and expose the ones that quietly lie.

1 · The data obstacle: usable 3D supervision, not just asset count

Recovery and generation look symmetric but are not, and much of the asymmetry is a data problem. To recover 3D you can measure geometry (lesson 03) or constrain a scene with several observations (lessons 09–12). To generate 3D you need a prior over plausible shape, appearance, topology, articulation, and often physics — including parts no camera saw. Large 3D collections exist, but their useful supervision is far less uniform than their object counts suggest.

Image-text pretraining can exploit enormous, weakly labeled web corpora. Objaverse-scale collections and captured multiview datasets have made native 3D training possible, but a model-ready asset may still need watertight or at least coherent geometry, consistent units and orientation, usable topology/UVs/materials, camera trajectories, licensing, semantic parts, articulation, collisions, and physical properties. Many files are duplicates, broken, view-biased, or missing those fields. The bottleneck is therefore not a timeless “1000×” ratio; it is the amount, diversity, and provenance of canonicalized, renderable, physically and semantically meaningful 3D for the task.

2 · Historical foundation — 2D-prior distillation (SDS / DreamFusion)

DreamFusion was a pivotal proof of concept: its per-prompt optimization does not require a paired text–3D training corpus. A pretrained text-to-image diffusion model (CV lesson 17) contains powerful appearance and semantic priors, though not a guaranteed coherent model of every viewpoint. The method asks a differentiably rendered 3D representation to find a shape whose random-view images the 2D prior considers plausible.

Set up the loop exactly as in lesson 01 §1, but with a generative critic in place of a captured image. Let the thing we optimize be a 3D representation with parameters θ — a NeRF (lesson 09) or a set of 3D Gaussians (lesson 10). Repeat:

• Sample a random camera pose around the object.
• Render the representation from that pose through the differentiable renderer (lesson 01 §4) to get an image x = g(θ).
• Ask a frozen pretrained diffusion model: "how would you nudge this image to make it a more plausible sample for the text prompt y?" That nudge is the score — the diffusion model's estimate of the direction of increasing likelihood — and it is available for free from the network's noise prediction.
• Backpropagate that image-space nudge through the differentiable renderer into the 3D parameters θ.

This is Score Distillation Sampling (SDS), the engine of DreamFusion. Concretely, noise the render, let the diffusion U-Net \hat{ε}_\phi predict the noise, and the SDS gradient of the loss L w.r.t. the 3D parameters is (dropping a per-step weight w(t)):

Read it piece by piece. The bracket (\hat{ε}_φ − ε) supplies an image-space update derived from the diffusion denoiser for prompt y. The factor ∂x/∂θ is the renderer Jacobian: it carries that update into 3D parameters. SDS avoids backpropagating the full diffusion-model Jacobian and uses the noise residual as a practical gradient estimator. The asset-optimization loop needs no paired 3D target; its knowledge still comes from the images, captions, biases, and any view-conditioning data used to train the prior.

Fixes, and why they work. Two lines of attack, each removing one root cause:

• Multiview-aware diffusion (MVDream). Fine-tune the diffusion model to generate several consistent views at once, so the prior is camera-aware rather than treating every view as an unrelated image. This reduces the structural cause of Janus, though consistency is not guaranteed.
• Variational SDS (ProlificDreamer). Replace the single-point-estimate SDS update with a variational objective (VSD) that models a distribution over renders rather than pushing toward one mode. This can reduce saturation and improve diversity at lower guidance.

3 · Native 3D generation

The native family confronts the data gap head-on: train a generative model directly on 3D data (Objaverse and friends). Diffusion, flow matching, or autoregression can operate over different 3D encodings. The first question is therefore what 3D representation do you generate?, and the answer recapitulates the representation zoo of lesson 01:

• Point clouds — diffuse or autoregress a set of surface samples (e.g. Point-E). Cheap, but without connectivity or an explicit continuous surface.
• Voxels — generate an occupancy/feature grid; simple but pays the O(N³) tax of lesson 01 §3.
• Meshes — autoregress vertices and faces directly (e.g. token-by-token mesh generation) for artist-ready output.
• Structured latents / tokens — encode fields, Gaussians, meshes, or parts into compact continuous latents or discrete tokens, then use diffusion, flow matching, or autoregression. The decoder determines whether the result is merely renderable or also editable, watertight, riggable, and simulation-ready.

An influential representation for 3D-aware generation is the triplane, popularized by EG3D. The obstacle is exactly lesson 01 §3: a dense 3D feature volume is O(N³) and blows up memory. The triplane factors that volume into three orthogonal 2D feature planes — one each in the XY, XZ, and YZ planes. To read the feature at any 3D point (x,y,z), project it onto all three planes, bilinearly sample each, sum (or concatenate) the feature vectors, and decode them into color, density, SDF, or another field quantity:

Three wins fall out at once. Memory drops from O(N³) to O(3N²). The planes are ordinary 2D feature maps, so mature 2D CNN/diffusion machinery applies. A small decoder preserves continuous queries and differentiable rendering. But this factorization is not free: axis-aligned planes impose representation bias, and a radiance-field triplane is not automatically an artist-ready mesh. Modern native systems also generate mesh tokens, sparse voxel/point latents, Gaussian sets, and part graphs; the output contract should select the representation.

4 · Multiview priors and feed-forward reconstruction (LRM)

SDS pays optimization cost per asset; a native generator samples from its learned 3D distribution. A Large Reconstruction Model (LRM) solves a different problem: it amortizes image→3D inference into a transformer trained on large multiview/3D datasets, mapping one or a few images directly to a 3D representation — a triplane, Gaussians, a field, or a mesh — in a forward pass. Some systems stop there; others spend a smaller test-time refinement budget to enforce the input views or improve topology.

This is the sharpest contrast in the whole track. Recall how lessons 09–10 produce a 3D asset: you take posed images of one scene and run gradient descent for minutes-to-hours on that scene's parameters, from scratch, every time. NeRF and 3DGS are per-scene optimizations — brilliant fits, but the work is not reusable across scenes. LRM amortizes that optimization into a network: it pays the cost once, at training time, over many scenes, and thereafter reconstruction is a forward pass. Put side by side:

The trade is prior strength and speed versus instance-specific fitting. A feed-forward model can plausibly complete unobserved regions but inherits its training distribution; per-scene optimization can fit the observed views more tightly but cannot recover unseen truth without a prior. For “here is a photo, give me a usable asset now,” the strongest practical pattern is increasingly hybrid.

The modern landscape is a pipeline, not three sealed boxes

1. Condition and expand the evidence. A multiview-aware image/video diffusion model produces a camera-indexed set of jointly consistent views, normals, or depth rather than scoring each view independently. This gives the reconstruction stage more evidence, but generated views are hypotheses — not measurements.
2. Reconstruct feed-forward. An LRM or geometry foundation model predicts Gaussians, a field, or a coarse mesh in one pass. This amortizes the expensive search and establishes a consistent coordinate frame.
3. Convert for the product. Extract or generate a mesh if collision, UV editing, rigging, or conventional rendering is required; keep Gaussians or a field if novel-view quality is the product.
4. Refine and validate. Optimize briefly against the real and generated views, regularize topology/materials, and reject cross-view contradictions. Refinement can improve consistency, but it cannot turn a hallucinated back view into ground truth.

Native 3D generation remains important when the desired output is itself structured — valid mesh topology, semantic parts, articulation, or scene graphs. Multiview generation is strongest when web-scale appearance priors matter. Feed-forward reconstruction is strongest when latency matters. A “state-of-the-art method” is therefore defined by the output contract and validation set, not by one universal architecture.

5 · Dynamics — the fourth dimension (4D)

Everything up to here, generation included, assumed a static scene. Real scenes move: people walk, cloth flaps, cars drive. Adding time turns 3D into 4D, and the representations of lessons 09–10 extend in two natural ways.

Dynamic NeRF / 4D Gaussian Splatting. Two strategies dominate, and they map onto the implicit/explicit split:

• Deformation field (implicit-leaning). Keep a single canonical radiance field (the object at rest) and learn a time-dependent warp D(x, t) that maps each point at time t back into canonical space before querying: c,σ = F_canonical(x + D(x,t)). Motion is factored out from appearance, which is compact and interpolates smoothly.
• Per-primitive trajectories (explicit-leaning). With 3D Gaussians, give each blob its own motion trajectory over time (its center, and often rotation/scale, as functions of t). The scene is then just the Gaussians of lesson 10, animated — which keeps 3DGS's real-time rendering while adding time.

Parametric models for the special case that matters most: humans. A general deformation field is unconstrained; but humans (bodies, faces, hands) are so common and so structured that the field built low-dimensional parametric mesh generators for them, fit from data:

Each is a differentiable function M(θ, β) → \text{mesh}: feed a few dozen numbers, get a full mesh. Crucially the rotations here are exactly the SO(3) joint rotations of lesson 02 — SMPL's pose θ is a stack of axis-angle rotations, optimized on the manifold. Because the parameter count is tiny, you fit the model to images or point clouds by optimizing those few parameters (the inverse-rendering loop of lesson 01, restricted to a low-dimensional prior). This is what powers markerless motion capture, avatar/AR try-on, and animation: the parametric model supplies the missing 3D prior that makes an otherwise ill-posed fit well-posed.

6 · 3D systems and deployment — the CV 19 analog

This section is to this track what CV lesson 19 was to the 2D track: the shift from "does the method work in a paper" to "does the system survive contact with a product." Three parts: representation by product, evaluation metrics and their pitfalls, and the silent failures nobody puts in the paper.

(a) Representation by product

The whole track's refrain — choose the representation from the constraint, not the fashion (lesson 01) — is decided here. Four products, almost no shared design choices:

Two of these deserve a note. Robotics leans on TSDF fusion (truncated signed distance function) — the KinectFusion recipe: integrate each depth frame into a running voxel SDF, averaging out sensor noise into a clean surface you can both render and query for collisions. And autonomous stacks increasingly predict occupancy directly (lesson 11) precisely because a bounding-box detector fails on the object classes it was never trained on, whereas "is this volume occupied?" degrades gracefully.

(b) Evaluation metrics — and their pitfalls

Each 3D task has its own metric; you must know what each measures and where each lies.

(c) The silent failures

These sink real 3D systems and appear in no benchmark:

• Calibration and extrinsics. Every multi-sensor rig needs each sensor's intrinsics and its pose relative to the others (the SE(3) extrinsics of lesson 02). If a LiDAR-to-camera transform is off, projected points land on the wrong pixels and fusion silently corrupts. Learned or optimization-based systems can estimate or jointly refine calibration when motion and scene constraints make it observable, but they cannot reliably compensate for arbitrary, drifting, or unobserved calibration errors. Calibrate, monitor, and version it as part of the sensor state.
• Time synchronization. A moving car at 15 m/s travels 15 cm in 10 ms. If the camera and LiDAR are not hardware-time-synced, a fast-moving object is in different places in the two sensors' data, and fusion smears it. Sync is a systems problem, not a learning one.
• Annotation cost. Labeling 3D is far more expensive than 2D — annotators draw boxes in BEV/3D and the labels must be propagated across frames of a sequence. This is why 3D datasets are small (the §1 problem again, now on the supervised side).
• Sim-to-real gap. Synthetic data can provide exact labels and rare interventions, but transfer depends on more than photorealism. The scene/actor distribution, action policy, physics, optics, sensor timing, noise, and label conventions can each differ from deployment. Close the loop with real-data calibration, domain randomization, targeted scenario generation, and held-out real evaluation.

Where this leaves you — the whole track in one line

Step all the way back. Lesson 01 introduced a powerful loop: pick a 3D representation → render or project it → compare to observations → update the representation. Across these thirteen lessons, lesson 02 gave the SE(3) machinery to place and optimize poses; lessons 03–05 moved from depth and point clouds through registration to surfaces; lessons 06–08 built the forward graphics path; lessons 09–10 built differentiable radiance-field and Gaussian renderers; lesson 11 learned across 3D scenes; and lesson 12 recovered relative, metric, calibrated, and unposed geometry from images.

This capstone changes the source of constraint. Reconstruction is anchored primarily by measurements; generation is anchored primarily by a learned prior; modern systems mix both. SDS literally pushes a prior through a renderer, LRMs amortize reconstruction, and native generators learn a distribution directly in a 3D representation — so not every method is the same inverse-rendering algorithm. The unifying questions are deeper: what is represented, what observations or priors constrain it, which transformations preserve consistency, where is computation paid, and how is failure measured? Dynamics adds time; deployment adds sensors and budgets; the synthetic-data bridge turns the forward engine into a controlled source of observations. That is the whole track, and it points back to the track index and its four questions — representation, rendering, recovery, system.

// nearest-neighbour distance from point p to set Q (brute force)
function nnDist(p, Q){
  var best = Infinity;
  for (var i=0;i<Q.length;i++){
    var dx=p.x-Q[i].x, dy=p.y-Q[i].y, d=dx*dx+dy*dy;
    if (d<best) best=d;
  }
  return Math.sqrt(best);
}
// directional Chamfer = MEAN of nearest-neighbour distances (one direction)
function chamferDir(A, B){
  var s=0; for (var i=0;i<A.length;i++) s+=nnDist(A[i], B); return s/A.length;
}
var cGP = chamferDir(GT, PRED);        // GT → pred
var cPG = chamferDir(PRED, GT);        // pred → GT
var symmetric = 0.5*(cGP + cPG);       // symmetric Chamfer (average form)

// F-score@τ = harmonic mean of precision and recall of points within τ
function frac(A, B, tau){               // fraction of A within tau of some B point
  var n=0; for (var i=0;i<A.length;i++) if (nnDist(A[i],B)<=tau) n++;
  return n/A.length;
}
var precision = frac(PRED, GT, tau);    // preds that are near a GT point
var recall    = frac(GT, PRED, tau);    // GT points that are covered by a pred
var F = (precision+recall>0) ? 2*precision*recall/(precision+recall) : 0;

// KEY: adding one far outlier to PRED spikes chamferDir(PRED,GT) (its NN dist is huge,
//   and it enters the MEAN) while precision drops by only 1/|PRED| → F-score barely moves.

Interview prompts

• What is the actual 3D-data bottleneck, and why are “exactly three routes” too simple? (§1 — raw asset count hides broken geometry, inconsistent scale/materials, missing cameras/semantics/physics, and licensing. 2D-prior distillation, native 3D, and feed-forward reconstruction are useful overlapping families; modern systems combine multiview generation, reconstruction, native tokens, and refinement.)
• Explain Score Distillation Sampling and the Janus artifact. (§2 — optimize a NeRF/3DGS so random-view renders please a frozen 2D diffusion prior, backprop through the differentiable renderer; per-view supervision biased to canonical fronts yields repeated faces — fix with multiview diffusion / variational SDS.)
• What is a triplane and what problem does it solve? (§3 — factor an O(N³) feature volume into three 2D planes + a tiny MLP; cuts memory to O(N²) and lets 2D CNN/diffusion backbones generate 3D-aware content.)
• Contrast an LRM with per-scene NeRF/3DGS optimization, then describe the modern hybrid. (§4 — per-scene methods fit observed views iteratively; LRMs amortize image→3D into a forward pass but inherit their training prior. A hybrid generates consistent views/depth/normals, reconstructs feed-forward, converts to the product representation, then refines and validates.)
• How do you add time to a NeRF or Gaussian scene, and how do you make an ill-posed human fit well-posed? (§5 — a deformation field warping to a canonical space, or per-Gaussian trajectories; fit a low-dimensional parametric model (SMPL/FLAME/MANO) whose few pose θ/shape β parameters supply the missing prior.)
• Your reconstruction has a good Chamfer distance but looks wrong — what happened, and what would you report? (§6 — nearest-neighbor averaging can hide topology/detail errors and can also be sensitive to distant outliers; state the squared/unsquared convention and sampling, add F-score at meaningful thresholds, and for novel views pair PSNR/SSIM with perceptual and task metrics.)