Flow matching — Euler sampling
A trained vθ gives you an ODE. Solve it. The step count is the whole story.
The ODE
A trained velocity field defines an ordinary differential equation: the trajectory of a particle that starts at x0 ∼ N(0, I) and rides the field. The IVP is
By construction (lesson 5), the resulting x(1) is a sample from pdata. To produce a batch of samples, integrate one ODE per sample — in batched matrix form, that’s one tensor of shape (N, D) updated in lockstep.
Forward Euler — the simplest integrator
Forward Euler turns the ODE into one update line:
For K steps total, that’s K evaluations of vθ. FlowMatching.sample:
x = randn(n, *shape)
dt = 1.0 / steps
for k in range(steps):
t = full((n,), k * dt)
x = x + dt * self.model(x, t)
return xThree lines. This is the payoff for the design effort — the sampler is trivial because the path is straight.
Claim: each Euler step is wrong by O(dt2), yet the error you actually see at the end is only O(dt) = O(1/K).
- One step assumes the velocity holds still. Euler reads the field once, at the start of the step, then walks in a straight line for the whole interval dt. But the true velocity keeps changing as you move — that’s the dt2/2 · ẍ term Taylor warns you about. The first term you drop is quadratic in dt, so one step is off by O(dt2).
- But you take many steps. To cross t ∈ [0, 1] you stitch together K = 1/dt of these little straight hops.
- Errors add up, so multiply. Roughly 1/dt steps, each costing O(dt2), gives a total of (1/dt) · dt2 = O(dt) = O(1/K). One power of dt cancels.
Central point. The end-to-end error shrinks only as fast as 1/K — linearly in the number of network calls — so doubling the steps roughly halves the error, and halving the steps roughly doubles it.
Why few steps suffice (for FM specifically)
Here is the intuition before the symbols: Euler walks in straight lines, so it only goes wrong where the true path bends. A perfectly straight trajectory can be traced exactly in a single step; a sharply curving one needs many tiny steps to follow the bend. So the question “how many steps do I need?” is really the question “how curved are the paths I’m integrating?” Formally, the off-path drift Euler accumulates is governed by the second derivative ẍ = ∂tv + v · ∇xv — the curvature of the integral curves. Big ẍ means lots of bending, which means lots of steps; small ẍ means you can get away with few.
And flow matching is built to make ẍ small. For the linear path, each conditional trajectory is exactly a line, so the conditional curvature is zero. The marginal trajectories (the curves you actually integrate at test time) have small curvature inherited from this. Empirically:
Claim: FM needs ~20× fewer steps than DDPM because the paths it asks you to follow are nearly straight.
- Euler’s only enemy is curvature. A straight-line step is a perfect approximation of a straight path and a bad one of a bending path. The more a trajectory curves, the smaller the steps you need to stay on it.
- The training recipe forces the building blocks to be straight. Flow matching defines each conditional path as xt = (1−t)·x0 + t·x1 — a literal straight line from noise to a data point. A straight line has zero curvature, so for those conditional paths there is nothing for Euler to miss.
- The paths you actually integrate inherit that straightness. At test time you ride the marginal field (the averaged-together version), which can bend a little where many straight conditional paths cross. But it starts from a stack of perfectly straight ingredients, so the bending stays mild.
- Mild bending ⇒ few steps. Since steps-needed tracks curvature, low curvature cashes out directly as a low step count — the table below: comparable quality at roughly 1/20th the network calls of DDPM.
Central point. DDPM’s sampling trajectories are curved (variance-preserving paths bow toward the origin), so it needs hundreds of steps; FM deliberately builds straight paths, so a handful of Euler steps already lands on the data.
| Solver | Steps (typical) | FID on CIFAR-10 (Lipman 2023) |
|---|---|---|
| FM Euler | 20–50 | < 5 |
| FM RK4 | 10 | < 5 |
| DDPM ancestral | 1000 | < 5 |
| DDIM | 50–100 | < 5 with caveats |
Roughly: same quality, ~20× fewer network calls.
Interactive · Euler on a learnable toy velocity field
The widget below trains a tiny MLP vθ on the two moons (50 seconds in JS), then integrates with Euler at K steps. Watch what happens at K = 2, 5, 50. With a straight path you see usable two moons even at K = 5.
3D · trajectories of your trained model
The widget above renders only the final samples. Here’s what the integrator is actually doing: 30 particles followed step by step through (x, y, t) space, with t as the third axis. Train the model in the widget above, then come here and hit render — straight-ish lines from a Gaussian cloud at t = 0 to the two moons at t = 1. The straighter the lines, the fewer integration steps needed.
RK4 in one line
If you want lower error per step at the cost of more network calls, use a higher-order Runge-Kutta. RK4 evaluates vθ four times per step:
k1 = v(x, t)
k2 = v(x + dt/2 * k1, t + dt/2)
k3 = v(x + dt/2 * k2, t + dt/2)
k4 = v(x + dt * k3, t + dt)
x = x + dt/6 * (k1 + 2*k2 + 2*k3 + k4)Global error: O(dt4). Quality gain isn’t free — 4× the compute — but for the same total budget you often want fewer-but-better steps. The widget above lets you switch.
The bigger picture: when to pick what
Common gotchas
- Integrating the wrong direction. Training is x0 → x1. Sampling starts at x0 and integrates forward in t. If you accidentally start at x1 and integrate backward, the field will push you towards N(0, I) — looks like the model learned to noise, not denoise. This is the single most common bug.
- Off-by-one in the time index. The k-th step uses t = k · dt, evaluated before the update. Putting t = (k+1) · dt is a half-step error that compounds.
- Forgetting to scale t for the embedding. Sinusoidal time embeddings are tuned for an integer range like [0, 1000]. With t ∈ [0, 1] the high-frequency entries collapse — solution is the
t * 1000trick inMLPVelocity.forward. Lesson 8 talks about this for DiT.