Adapt & control a frozen model

You now have a trained multi-GB Stable Diffusion (lesson 22). You can’t retrain it for every new face, style, or spatial constraint. Every method in this lesson answers one question: where do you inject new information into a frozen network?

The one axis

Interview prep lists LoRA, DreamBooth, Textual Inversion, HyperNetworks, ControlNet, IP-Adapter, inpainting as separate techniques. They are points on a single spectrum — how deep into the frozen denoiser you reach, and how many parameters you pay:

Inject at the embedding: Textual Inversion

The lightest touch. Freeze everything — UNet, VAE, text encoder. Add one new token “S*” to the vocabulary and learn only its embedding vector (a few KB) so that “a photo of S*” reconstructs your 3–5 example images, using the ordinary diffusion loss (lesson 3) as the training signal. You are searching the text-embedding space (lesson 20) for the point that means your concept.

Pro: tiny file, composes with other tokens, model untouched.
Con: limited capacity — one vector can only describe what the frozen model can already render; it can’t teach genuinely new visual detail.

Inject as weight deltas: LoRA

The one-line description (“add a low-rank matrix to frozen weights”) is exactly right; here is why it works. A weight update ΔW for fine-tuning is empirically low-rank — adaptation lives in a small subspace. So instead of learning the full d×d matrix, learn two thin factors:

W' = W + ΔW = W + B A, A ∈ ℝ^r×d, B ∈ ℝ^d×r, r ≪ d

Read A and B as a two-step funnel: A squeezes the input down to r numbers (the down-projection), B fans those r numbers back up (the up-projection). Because the signal has to pass through that r-wide waist, the whole update BA can only push the weights in r directions — that is what “rank r” means. Freeze W; train only A and B, with B=0 at init so the product BA starts at zero and training begins exactly at the pretrained model. The parameter count drops from d² (the full matrix) to 2rd (the two thin factors) — for d=1024, r=8 that’s a 64× reduction, a few MB. And since BA is just another d×d matrix, at inference you fold it back into W once, so there is zero added latency. In SD, LoRA is applied to the cross- and self-attention projection matrices (lesson 20) — exactly where style and concept live.

Intuition · linear unpacking

Claim: a thin rank-r product can stand in for a full d×d fine-tuning update, at a fraction of the cost.

The bill for the full update. Re-learning the whole matrix means storing and training d² numbers per layer — for big d that is millions of params, a full model’s worth, plus the risk of overwriting what the model already knew.
Adaptation is small. Empirically, going from “generic model” to “your style” nudges the weights along only a handful of directions, not all d of them. The change you actually need is low-rank — it fits in a small subspace.
So buy only those directions. Writing ΔW = BA with an r-wide waist forces the update into exactly r directions. You pay for 2rd numbers instead of d² — the funnel, not the full matrix.
Free at inference. The two factors multiply back into one ordinary d×d matrix you add to W once, so the running model is the same size and speed as before — the savings are pure, not a runtime tax.

Central point. The expensive thing (a full weight rewrite) and the thing you actually need (a nudge in a few directions) are not the same size — LoRA only pays for the second.

Why low-rank is enough

Picture the pretrained weights as a finished sculpture. Full fine-tuning lets you re-carve every surface (powerful, but you need the whole block of marble = full params + risk of forgetting). LoRA says: most edits are a thin layer of clay added on top in a few directions. r is how many directions you allow. Bigger r = more expressive adapter, more params, more overfit risk — the one knob you tune.

Inject extra attention: IP-Adapter (image prompts)

Goal: use an image as the prompt. The naive route — concatenate CLIP image features with text features and feed the existing cross-attention — works poorly, because one set of K/V weights can’t serve two very different modalities. IP-Adapter (Ye et al. 2023) uses decoupled cross-attention: keep the text cross-attention untouched, and add a parallel cross-attention layer with its own W_K, W_V for the image features, then sum the two attention outputs. Only the new image-branch K/V projections train (~22M params); the base model is frozen.

out = Attn(Q, K_text, V_text) + λ · Attn(Q, K_img, V_img)

This is the same “don’t fight the existing pathway, add a parallel one” instinct as LoRA, applied at the attention level instead of the weight level.

Inject generated layers: HyperNetworks

A HyperNetwork is a small network that outputs weights to insert into the main model’s layers — you train the generator of the adapter, not the adapter directly. In the SD community it usually means small networks that transform the attention K/V projections toward a target style. Conceptually it sits between LoRA (fixed delta) and full fine-tuning (it can produce input-conditioned deltas), but it has largely been superseded by LoRA, which is simpler and folds into the weights for free.

Inject a parallel branch: ControlNet (spatial control)

The previous methods change what is generated; ControlNet (Zhang et al. 2023) controls where — condition on an edge map, depth map, pose, scribble. The construction is elegant:

Clone the UNet’s encoder + middle block (structure and weights). Freeze the original; train the copy.
Pass the conditioning image c through a small trainable conv encoder to match the latent’s shape, add that to the copy’s input, and Add each block’s output back into the corresponding layer of the frozen decoder.
Connect the copy through zero-convolutions — 1×1 convs initialized to zero weight and bias.

Why zero-convolution is the whole trick

At step 0, the zero-convs output zero, so the ControlNet branch contributes nothing — the model is byte-for-byte the original SD, producing identical output. Training starts from a guaranteed-no-harm state; gradients flow through the zero-conv (its input is nonzero, so ∂loss/∂weight ≠ 0), and the branch gradually learns to inject the spatial signal. No warmup instability, no risk of destroying the base model on step one. It is the adaLN-Zero idea from lesson 8 (“start at identity”) applied to a whole side branch.

Intuition · linear unpacking

Claim: a connection that starts at zero lets you bolt a whole new branch onto a trained model without ever risking the model it was bolted onto.

The danger of grafting. A freshly-initialized control branch outputs garbage at first. If you wired it straight into the frozen UNet, that garbage would corrupt the decoder on step one — you could wreck a model that took weeks to train before learning anything.
Zero-init mutes the graft. A 1×1 conv with zero weights multiplies everything it receives by zero, so on step 0 the branch adds exactly nothing. The combined network is byte-for-byte the original SD — guaranteed no harm.
But it isn’t stuck at zero. The gradient of a weight is its input times the upstream signal; the input here is nonzero (real conditioning flows in), so ∂loss/∂weight ≠ 0 even though the output is zero. The conv can learn its way off zero.
It opens the valve slowly. As those weights grow from zero, the branch contributes a little more spatial signal each step — control fades in from “none” rather than slamming on. No warmup instability, no destructive first step.

Central point. Starting the connection at zero makes “add a new branch” begin as a no-op and become useful only as fast as it can be learned safely.

Inject all weights: DreamBooth

The heavy end: fine-tune the entire UNet on your 3–5 subject images, binding them to a rare token (“a [V] dog”). Highest fidelity for a specific subject — the familiar “best results, biggest storage” trade (a full model copy). Two failure modes it must defend against:

Overfitting / language drift — with 5 images the model forgets what “dog” means in general. DreamBooth’s fix is a prior-preservation loss: also train on the model’s own “a dog” samples so the class concept is retained while the [V] token specializes.
Cost — a full checkpoint per concept. In practice people run DreamBooth through a LoRA (“DreamBooth-LoRA”): the subject-binding training of DreamBooth, but the only trainable params are a LoRA — best of both ends of the axis.

Inpainting: control by latent surgery, no new params

Editing a region needs no training at all. Given a mask, at every denoising step replace the known (unmasked) region of the latent with the correctly-noised original, and let the model only generate inside the mask:

x_t−1 = mask ⊙ (denoiser step) + (1−mask) ⊙ (forward-noised original to level t−1)

The unmasked context is always “correct,” so cross-attention and self-attention (lesson 20) make the in-painted region blend with it. (Dedicated inpainting checkpoints add the mask as an extra UNet input channel for cleaner seams, but the blending recipe alone already works.) It is the same “known region stays fixed” trick that powers ControlNet-style consistency, done in latent space.

Interactive · adapter cost vs. reach

Each method as a point: trainable parameters (log scale) vs. how deep it reaches into the frozen model. Click one to see what it injects, where, and its tradeoff. The lesson: there is a Pareto frontier — tiny-and-shallow (TI) up to heavy-and-deep (DreamBooth) — and LoRA sits at the knee, which is why it won.

Punchline · end of Part C

Seven “different” techniques, one question: where do you inject? Embedding (Textual Inversion) → low-rank weight deltas (LoRA) → parallel attention (IP-Adapter) → generated layers (HyperNetwork) → zero-conv parallel branch (ControlNet) → all weights (DreamBooth) → latent surgery, no params (inpainting). Shallower = cheaper and more composable; deeper = higher fidelity and more storage. LoRA won the everyday case by sitting at the knee of that curve.

End of series

Twenty-three lessons. Part A derived continuous diffusion and flow matching from scratch; Part B took the discrete/multimodal fork; Part C returned to build the production text-to-image stack — score theory, samplers, guidance, latent space, CLIP, evaluation, the SD lineage, and the adapter ecosystem. Every applied trick here reduced to a first principle from Parts A–B. That was the point.