The production lineage: SD → SDXL → SD3
Lessons 16–21 built every part separately. Here they snap together into the actual product, and we trace what each generation changed and why — plus how DALL·E took a different road to the same goal.
Stable Diffusion = five lessons assembled
There is no new idea in SD v1.5 — it is the parts you already have, bolted together. That is worth seeing explicitly, because the stock “what is SD’s core optimization / how is SD trained” questions are just “name the parts.”
| Component | What it is | Lesson |
|---|---|---|
| VAE (encoder + decoder) | compress 512×512×3 → 64×64×4 latent; diffuse there, decode once | 19 |
| UNet denoiser (ε-prediction) | the score estimator, with self- and cross-attention blocks | 2–3, 16 |
| CLIP text encoder (frozen) | prompt → per-token embeddings, fed as cross-attention K/V | 20 |
| Classifier-free guidance | 10% caption dropout in training; w·(cond−uncond) at sampling | 18 |
| Sampler (DDIM / DPM-Solver) | integrate the PF-ODE in 20–50 steps | 17 |
The training loop (“how is SD trained?”) is lesson 3’s four-liner with two wrappers: encode the image to a latent first, and pass the CLIP text embedding into the UNet:
# one SD training step
z0 = vae.encode(image).sample() * 0.18215 # to latent, rescale to ~unit var
cap = clip_text(prompt) # frozen; 10% of the time → NULL
t = randint(0, T, (B,))
eps = randn_like(z0)
zt = sqrt(abar[t]) * z0 + sqrt(1-abar[t]) * eps # forward, Eq. ⋆ (lesson 2)
eps_pred = unet(zt, t, cap) # cross-attn on cap
loss = ((eps - eps_pred) ** 2).mean() # L_simple (lesson 3)SD vs. DALL·E: two roads, same destination
A classic interview question asks for the similarities and differences between SD and DALL·E. They differ in how the prompt becomes pixels:
| Stable Diffusion | DALL·E 2 (unCLIP) | |
|---|---|---|
| Text → image bridge | CLIP text embedding → cross-attention, directly conditions diffusion | a prior maps CLIP-text → CLIP-image embedding, then a diffusion decoder renders it |
| Where generation happens | latent diffusion (VAE space) | diffusion decoder in pixel/upsampled space |
| DALL·E 1 (for contrast) | — | fully autoregressive: a transformer predicts dVAE image tokens one at a time |
So “SD vs DALL·E” spans both axes the series has drawn: latent diffusion vs. autoregressive tokens (Part A vs. Part B) and direct conditioning vs. a CLIP-embedding prior. DALL·E 3 later moved toward SD-style diffusion with a heavily-captioned training set and an LLM prompt-rewriter (lesson 15’s hybrid).
SDXL: scale the same recipe
SDXL (Podell et al. 2023) keeps the LDM recipe and pushes on engineering. The change-list is the usual interview fodder; here is the why for each:
- ~3× bigger UNet — more capacity for 1024² generation.
- Two text encoders (CLIP-L + OpenCLIP ViT-bigG), embeddings concatenated — more text capacity while staying CLIP-native (lesson 20’s “more encoders, complementary biases”).
- Base + refiner — a second LDM trained only on low noise levels (last ~200 steps). The base lays out structure; the refiner, run on a partly-noised version of the base latent, sharpens detail. It is a specialist for the easy end of the trajectory.
- Size & crop conditioning — tell the model, for each training image, how big the original was and where it was cropped from (the resolution and crop offset, Fourier-encoded and added to the time embedding). The point: a big web image set is full of small thumbnails and off-center crops, and naively training on them teaches the model to produce blurry, half-cut objects. By handing the model those facts as a label, you let it use the bad images for learning while keeping the badness attributable to a knob you control. At inference you simply set the knob to “full size, crop (0,0)” and ask for the clean behavior. A clever fix for a data-curation problem.
- Multi-aspect-ratio training — bucket images by aspect ratio so the model can produce 16:9, 3:4, etc., not just squares.
Claim: conditioning on size and crop lets SDXL train on junk images without inheriting their junk.
- The bind. Web-scale data is messy — lots of tiny upscaled thumbnails and lazily off-center crops. Throw them away and you lose most of your data; keep them and the model learns that “an image” means blurry and half-cut.
- The move. Don’t hide the flaw — name it. Stamp each training example with its true resolution and crop offset and feed those numbers in alongside the timestep.
- Why that helps. Now the model doesn’t have to guess why an image is blurry or cropped; the label already says so. It learns “blurriness lives on the size knob, off-centering lives on the crop knob” instead of baking those defects into its idea of a normal picture.
- The payoff. At generation time you set the knobs to the ideal — full resolution, crop (0,0) — and the model gives you sharp, centered objects, even though it never saw many such images in training.
Central point. Telling the model what’s wrong with each training image turns a defect it would otherwise absorb into a dial you can turn off at inference.
SD3: switch the engine to rectified flow
SD3 (Esser et al. 2024) is where Part A’s “DDPM = flow matching” thread (lessons 5–7) pays off in a shipped model. Three changes:
- Rectified flow objective — replace the DDPM noising path with a straight conditional path (lesson 5’s flow matching with the linear interpolation xt = (1−t)x₀ + t·ε). The network regresses the constant velocity ε − x₀ along that line — related to, but not the same parameterization as, Salimans-Ho v-prediction. Straight paths integrate in far fewer steps (lesson 6) and the target is well-behaved at both endpoints (lesson 9).
- MM-DiT backbone — the multimodal DiT of lesson 15: text and image tokens share one bidirectional attention pool (“joint attention”), with separate weights per modality. Replaces SD’s convolutional UNet entirely.
- Three text encoders (CLIP-L + CLIP-G + T5-XXL) — T5 supplies the compositional / syntactic understanding CLIP lacks (lesson 20), so “A but not B” and text rendering improve. Released as a family from ~800M up to an 8B flagship (the shipped “Medium” is ~2B).
Claim: training the model to walk a straight line from noise to data is what lets SD3 sample in few steps.
- What changed. DDPM’s path from clean image to noise is a curved, schedule-shaped detour. Rectified flow instead picks the simplest possible path: the straight segment xt = (1−t)x₀ + t·ε that connects the data point to the noise point.
- Straight means constant speed. On a straight line, the direction never changes — the velocity is the same vector ε − x₀ the whole way. So the network has an easy, well-behaved thing to predict: one steady heading, not a target that swings wildly near the endpoints.
- Why fewer steps. Sampling means numerically following that velocity field. Following a curve accurately needs many tiny steps (a big step cuts the corner and drifts off). Following a near-straight line, you can take a few big steps and still land close — this is the lesson-6 step-count payoff.
Central point. Straightening the noise-to-data path makes the thing the model must follow nearly a ruler line, and a ruler line can be traced in a handful of strides instead of fifty.
Interactive · the SD inference pipeline, timed
Walk a prompt through the pipeline and watch where the wall-clock goes. Toggle latent vs. pixel diffusion (the lesson-19 speedup), change step count (lesson 17), and toggle CFG (which doubles UNet calls, lesson 18). The bar shows the dominant cost — and why “diffuse in latent space, distill the sampler” is the whole production-latency story.