Volumetric rendering and NeRF

Sample weight and color. The discrete analog of the termination density T(t)σ(t) is the weight

State plainly what this is: it is exactly front-to-back alpha compositing — the "over" operator, the alpha-blend the CV survey named without deriving. Walk the ray from the camera outward; at each sample you deposit its color scaled by how opaque it is (α_i) and by how much light is left to deposit it (T_i), then you attenuate the remaining budget by (1-α_i). Each T_i = \prod_{j is precisely the "remaining transparency" a painter tracks when layering translucent glazes front to back.

Two free readouts. Because the weights w_i are a distribution over termination depth, the same sum gives geometry for free. The expected depth (a depth map straight out of the field) and the accumulated opacity (how much the ray hit anything at all) are

Opacity near 0 means the ray sailed through empty space (useful for compositing NeRFs onto backgrounds); the expected depth is what the widget below plots as a vertical marker. All of this — color, depth, opacity — is differentiable in σ_i and \mathbf c_i, hence in the MLP weights, which is the entire reason we chose fog.

4 · Making the MLP actually work

The integral is correct but a naïve MLP renders blurry mush and cannot represent view-dependent effects. Four ideas turn the recipe into NeRF.

(a) Positional encoding — beating spectral bias

Feed raw coordinates (x,y,z) to an MLP and the output is smooth and blurry, missing all fine texture. The cause is spectral bias: MLPs preferentially fit low frequencies first and are strongly reluctant to represent high-frequency detail — the same frequency-domain intuition developed in CV lesson 02, now biting the network rather than a convolution. The fix is to lift each scalar input through a bank of sinusoids at geometrically increasing frequencies before the MLP sees it:

Applied to each of x,y,z (and, at lower L, to the direction), this injects high-frequency content into the input so that a smooth function of γ(p) can be a high-frequency function of p. Fine detail — bark, text, thin structures — becomes representable. Concretely with L=10 for position, a 3-vector becomes a 3 × 2 × 10 = 60-dimensional encoded input.

(b) View dependence — a deliberate inductive bias

Reiterating §1 now that we see it operationally: the MLP first maps encoded position to density σ and a feature vector; only then is the encoded direction concatenated to produce color \mathbf c. Density literally cannot see the direction. This architectural split is an inductive bias — it forces multiview-consistent geometry (one density field for all cameras) while still permitting view-dependent shading (color is free to swing with \mathbf d). It is the network-level enforcement of the physical argument from §1.

(c) Hierarchical sampling — spend samples where the scene is

Uniform samples along a ray waste almost all their budget in empty space, since a scene is mostly air with a thin shell of matter (the O(N^2)-surface-in-O(N^3)-volume observation from lesson 01 §3). NeRF trains two networks. A coarse network is queried at uniform samples and produces weights w_i. Normalize those weights into a piecewise-constant pdf \hat w_i = w_i / \sum_j w_j along the ray, then importance-sample a second, fine network from that pdf (inverse-CDF sampling), concentrating new samples exactly where the coarse pass thinks a surface is. The fine render uses the union of coarse and fine samples. Same integral, far fewer wasted evaluations.

(d) Training — geometry emerges from photometric consistency

The supervision is startlingly simple. From a set of posed images — camera intrinsics and SE(3) extrinsics recovered by SfM (CV lesson 05), the pose machinery of lesson 02 — cast a ray through each training pixel, render its color \hat{\mathbf C} by the sum in §3, and minimize the photometric mean-squared error against the observed pixel color:

There is no 3D supervision — no depth, no mesh, no occupancy labels. The density field is never told where surfaces are. Geometry emerges as the only configuration of fog that, composited by the volume rendering integral, reproduces all the training views simultaneously: a surface must sit where many rays from different cameras agree on color, because only a consistent opaque structure explains that agreement. This is multiview photometric consistency doing the work classical stereo did, but end-to-end differentiable. Optimization is per scene (the weights encode one scene) and takes minutes to hours on a GPU.

5 · Failure modes and speedups

NeRF is a landmark, not a finish line. Its limitations map directly onto later lessons:

The floater deserves a look because the widget makes it visible: a spurious blob of density floating in front of the true surface. The volume rendering integral handles it correctly in the sense that the front blob occludes what is behind it — transmittance T drops after the blob, so the true surface receives less weight and the expected depth is pulled forward. That is the mechanism, and it is also why floaters are pernicious: once one forms, it steals weight from the real geometry.

Speedups. The slowness is almost entirely the "hundreds of MLP queries per ray" cost, so the fixes attack the MLP:

• Instant-NGP — multiresolution hash grid. Replace most of the MLP's job with learned features stored on a grid, indexed at multiple resolutions through a spatial hash, feeding a tiny MLP. Because most cells are empty, the hash lets many grid coordinates share table slots without an explicit octree — the sparsity trick of lesson 01 §3 applied to a radiance field. Training drops from hours to seconds.
• Factorized fields (TensoRF). Represent the volume as a sum of low-rank outer products of vectors/matrices along the axes, turning a dense 3D grid into a few 1D/2D factors — compact and fast.
• Baking to real-time structures. Distill a trained NeRF into an explicit format (sparse voxels, textured meshes, or precomputed samples) that renders in real time.

Each of these keeps the volume rendering integral of §2–§3 and speeds up the field query. Lesson 10 takes the more radical step: abandon the implicit field and the per-ray march entirely, keep only the compositing equation, and composite millions of explicit Gaussian primitives that a GPU rasterizes in real time.

// two Gaussian density bumps along the ray; `sharp` narrows them (hard surface) vs wide (fog)
var sig = new Float64Array(N), col = [];       // density and emitted colour per sample
for (var i=0;i<N;i++){
  var t  = i/(N-1);                            // normalised depth in [0,1]
  var g1 = h1*Math.exp(-0.5*Math.pow((t-p1)/wdt,2));   // orange bump
  var g2 = h2*Math.exp(-0.5*Math.pow((t-p2)/wdt,2));   // teal   bump
  sig[i] = g1 + g2;
  col[i] = blend(ORANGE, g1, TEAL, g2);        // colour = density-weighted mix of the two
}

// the discrete volume-rendering / alpha-compositing sum (front to back)
var T = 1, C = [0,0,0], opac = 0, depth = 0, peak = 0;
for (var i=0;i<N;i++){
  var a = 1 - Math.exp(-sig[i]*d);             // α_i = 1 − e^(−σ_i·δ)
  var w = T * a;                               // w_i = T_i · α_i   (T_i = Π_{j<i}(1−α_j))
  C[0]+=w*col[i][0]; C[1]+=w*col[i][1]; C[2]+=w*col[i][2];
  opac += w;  depth += w * (i/(N-1));  peak = Math.max(peak, w);
  T *= (1 - a);                                // attenuate remaining light
}
// opac = Σ w_i = 1 − T_final ;  depth = Σ w_i t_i (expected termination depth)

Where this points next

We now have the derivation the CV track only surveyed: a scene is a continuous radiance field, a pixel is the expectation of emitted color over where its ray terminates, and evaluating that expectation is front-to-back alpha compositing — differentiable, so plain photometric gradient descent recovers geometry with no 3D labels. The one dissatisfaction left is speed: hundreds of MLP queries per ray is inherently slow, and the fixes in §5 chip at it without changing the shape of the computation. Lesson 10 keeps the compositing equation of §3 verbatim but throws out the implicit field and the per-ray march: it represents the scene as millions of explicit 3D Gaussians — the hybrid primitive introduced in lesson 01 — projects them to the image, sorts them by depth, and composites them with a GPU rasterizer, reaching real-time rendering while staying differentiable. NeRF taught us what to composite and why it trains; splatting changes only who is on the composite list.

Interview prompts

• Derive the volume rendering integral from the transmittance ODE, and interpret T(t)σ(t) probabilistically. (§2 — dT/dt=-σT ⇒ T=\exp(-\!\int σ); T(t)σ(t)\,dt is the probability the ray terminates in [t,t+dt), so \mathbf C is the expected color at termination.)
• Show the discretized integral equals front-to-back alpha compositing. (§3 — α_i=1-e^{-σ_iδ_i}, T_i=\prod_{j, \mathbf C=\sum_i T_iα_i\mathbf c_i; it is the "over" operator, identical to lesson 10's compositing.)
• Why does density depend on position only while color depends on position and direction? (§1, §4 — position-only density enforces multiview-consistent geometry; view-dependent color captures specular highlights. It is a deliberate inductive bias.)
• Why is positional encoding necessary, and what does it fix? (§4 — MLPs have spectral bias and fit low frequencies first; γ(p) injects high-frequency sinusoids so fine detail becomes representable — the CV lesson 02 frequency view.)
• How does NeRF recover geometry with no 3D supervision? (§4 — photometric MSE between rendered and observed pixels from posed (SfM) images; a consistent density field is the only fog that explains all views, so geometry emerges.)
• Why is vanilla NeRF slow, and how do Instant-NGP / TensoRF / splatting speed it up? (§5 — hundreds of MLP queries per ray; hash grids + tiny MLP, low-rank factorization, or baking; splatting drops the field and per-ray march, keeping only compositing.)