Part V — Fine-tuning

LoRA — parameter-efficient fine-tuning

Lesson 13 turned the base completer into an instruction-following assistant by fine-tuning it — but that meant nudging all ~124 million weights and then saving an entire second copy of the model just for that one behavior. Most of that update is redundant. This lesson replaces the whole-model update with a tiny low-rank patch that trains a fraction of a percent of the parameters, starts exactly at the base model, and can be swapped per task.

1 · The true cost of full fine-tuning

A single weight matrix in a transformer block — say a query projection W_q — is (d × d). For GPT-2 small that is 768 × 768 ≈ 590{,}000 numbers in one matrix, and a block has several such matrices (the four attention projections plus the two feed-forward matrices), stacked L = 12 times. Full fine-tuning computes a gradient for every one of those ≈124M numbers and writes back a new value for each.

Two costs follow, and the second is the one that bites. The compute cost is a full backward pass — real, but you pay it once. The storage and deployment cost is forever: the output of fine-tuning is a complete, standalone 124M-parameter checkpoint. Ten fine-tuned tasks means ten full checkpoints, ten times the disk, and — if you want to serve them — either ten models loaded in memory or an expensive swap of the entire weight set every time a request needs a different skill.

2 · The insight: the change you need is low-rank

Here is the observation that unlocks everything. When you adapt a large pretrained model to a specific downstream task, the update ΔW that fine-tuning discovers does not need the full expressive range of a (d × d) matrix. Empirically it has a very low intrinsic rank: it can be approximated well by a matrix that is the product of two thin ones. The pretrained W already carries the heavy, general knowledge; task adaptation only has to steer within a small subspace.

Recall what rank means. Any (d × d) matrix of rank r can be written as a product of a tall matrix and a wide one:

The tall A and wide B meet at a narrow waist of width r — the rank — that pinches all the information passing through. A full-rank update would need r = d and buy nothing. But if adaptation truly lives in a low-dimensional subspace, a small r (LoRA's paper uses values like 1, 2, 4, 8) recovers most of the benefit while the two factors together hold only 2·d·r numbers — vastly fewer than the d² in the dense update. This is the entire idea of Low-Rank Adaptation.

3 · The mechanics: freeze, patch, and start from zero

LoRA changes three things about the layer and nothing else. First, freeze W: the pretrained matrix keeps its values and receives no gradient. Second, add the low-rank patch beside it. The layer that used to compute Wx now computes the frozen result plus a small correction routed through the bottleneck:

Read the correction right-to-left, and notice we never build the big (d × d) matrix A·B explicitly. The input x (dimension d) first hits B and is squeezed down to the tiny rank-r vector Bx; then A lifts it back up to dimension d; finally the α/r factor scales it and it is added to the frozen Wx. Only A and B — together 2·d·r parameters — carry gradients and get trained.

Initialize B = 0 so training begins at the base model

The third change is the quiet, clever one. Initialize A with small random values but set B = 0. Then at step zero:

The very first forward pass is identical to the untouched base model — the patch adds nothing. This matters: fine-tuning should start from the model you already trust and move away from it deliberately, not lurch to a random new function on step one. As training proceeds, gradients flow into B (and A), the product A·B grows away from zero, and the model smoothly departs from the base only as far as the task demands. Setting A = 0 instead of B would also give ΔW = 0 at init, but then B would receive no gradient on the first step; zeroing B and keeping A nonzero keeps both factors learning immediately.

The α / r scale

The factor α/r decouples "how hard the patch pushes" from "how wide the bottleneck is." α is a fixed constant (often set once, e.g. to the initial r); dividing by r means that when you later raise the rank, the patch's overall magnitude does not blow up in proportion — you can retune r without re-tuning the learning rate. In practice α/r behaves like a second, LoRA-specific learning-rate multiplier on the update.

class LoRALinear:                 # wraps a frozen linear layer W: (d, d)
    def __init__(self, W, r, alpha):
        self.W = freeze(W)        # pretrained; no gradient
        self.A = zeros(d, r)      # will hold small random init...
        self.B = zeros(r, d)      # ...but B stays 0 at start  => ΔW = 0
        init_small_random(self.A)
        self.scale = alpha / r

    def forward(self, x):
        return x @ self.W.T + self.scale * ((x @ self.B.T) @ self.A.T)
        #      └ frozen base ┘         └ low-rank patch through the r-waist ┘

Everything downstream is unchanged: the loss is still cross-entropy over the response tokens (13), the optimizer is still AdamW (10). The only difference is which parameters carry gradients — and there are now dramatically fewer of them.

4 · The payoff: the arithmetic and the swap

Put GPT-2's numbers in. A dense (d × d) matrix with d = 768 holds 768² ≈ 590{,}000 parameters. Adapt it with rank r = 8 and the LoRA factors hold 2 · 768 · 8 ≈ 12{,}300 — about 2% of the dense matrix, a roughly 48× reduction, per matrix. Applied across the model, LoRA typically trains well under 1% of the total parameters while matching full fine-tuning's task quality closely.

The storage win compounds into a deployment win. Because W is frozen and shared, every task's adapter is just its own small A, B pair — a few megabytes instead of a few hundred. You load the one base model once, then hot-swap adapters per request: attach the summarizer's (A, B) for one call, the support bot's for the next, all over the same frozen backbone. Ten skills cost one model plus ten tiny patches, not ten models. (If you ever want a single fused checkpoint, you can add (α/r)·A·B back into W once and ship a plain model — but the whole point is usually to keep the base pristine and the patches separate.)

5 · Where to attach it, and rank as the capacity knob

You do not have to LoRA-ize every matrix. The original work found that adapting the attention projections — the W_q, W_k, W_v (and sometimes the output projection W_o) inside multi-head attention (06) — gives most of the benefit, since attention is where the model decides what to relate to what for the new task. The large feed-forward matrices can be left frozen, or included when you want more adapter capacity. This selectivity is part of why the trainable-parameter count stays so small.

The one knob that matters is the rank r. It is a direct dial on the adapter's capacity: too small and the low-rank patch cannot express the adaptation the task needs (it underfits, and quality lags full fine-tuning); larger and it captures more, at the cost of more trainable parameters and a weaker regularizing effect. Because init keeps you at the base model and α/r keeps the scale sane, sweeping r is cheap — which is exactly what the widget lets you do.

6 · Watch the parameter count collapse

The widget draws the frozen weight matrix W (greyed — it never changes) beside the two LoRA factors A and B that meet at the rank-r waist. Move the r slider to widen or pinch that waist, and the d slider to resize the whole layer. The KPIs compute the trainable parameter count 2·d·r against the full d², the percentage, and the compression factor — live. It shows B = 0 at initialization, so ΔW = A·B = 0: the patch starts invisible and the model starts exactly at the base.

Failure modes & checklist

Checkpoint

Where this points next

Step back. Over fourteen lessons you have built the whole machine from a single character: text became BPE ids, ids became embeddings with positions, a sliding window supplied free targets, attention was derived in three forced steps, the transformer block made depth trainable, a stack plus a head produced text, cross-entropy gave it a notion of wrong, AdamW drove it down, sampling and pretrained weights gave it fluency, and fine-tuning — classification, then instruction, now the cheap low-rank variant — bent it to a task. Every piece was a forced move, and every piece ran at a 124M toy scale. So the last question is the obvious one: what actually changes when you take this exact machine to a frontier model? The architecture, it turns out, barely does. Lesson 15, The whole machine — and what scale changes, replays the entire chain end to end and names precisely where scale lives — data, compute, systems, and the modern component swaps — before handing you off to the tracks that build there.

Interview prompts

• Why is full fine-tuning expensive even though the compute is a one-time cost? (§1 — the output is a complete standalone checkpoint per task, so storage and deployment cost scale linearly with the number of tasks.)
• What is the key empirical assumption LoRA relies on? (§2 — the adaptation matrix ΔW has low intrinsic rank, so it is well-approximated by a product of two thin matrices.)
• Write the LoRA forward pass and say which parts train. (§3 — y = Wx + (α/r)·A(Bx); W is frozen, only A and B (2·d·r params) train.)
• Why initialize B = 0, and why not A = 0 instead? (§3 — B=0 makes ΔW=0 so training starts at the base model, while A being nonzero lets both factors get gradients on step one; A=0 would starve B of gradient initially.)
• What does the α/r scale do? (§3 — it decouples the patch's magnitude from the rank, so you can retune r without re-tuning the learning rate; it acts as a LoRA-specific step multiplier.)
• For d=768, r=8, how many parameters does LoRA train per matrix versus full fine-tuning, and what is the ratio? (§4 — 2·768·8 ≈ 12{,}300 vs 768² ≈ 590{,}000; ratio 2r/d ≈ 2%, about 48× fewer.)
• Where do you attach LoRA and what is the single capacity knob? (§5 — typically the attention projections W_q, W_k, W_v (± W_o); the rank r is the capacity dial — too small underfits, larger costs more parameters.)

Companion reads: cs336 · 16 Supervised fine-tuning covers LoRA and QLoRA at systems scale; gpt_mini · 03 SFT shows fine-tuning as runnable post-training code.