Part V — Fine-tuning

Instruction fine-tuning — completer to assistant

Lesson 12 repurposed the base model for one fixed decision: swap the language-model head for a two-class head, read the last token, and you have a spam detector. But a classifier answers exactly one question. To get a model that follows arbitrary instructions — summarize this, translate that, rewrite this politely — we teach the base completer to treat "instruction → response" as the pattern to continue. This lesson builds the prompt template, prepares instruction/response data, and masks the loss to the response so the model learns to answer rather than to parrot the question.

What the last step left broken

Classification fine-tuning solved a single, narrow task by bolting on a task-specific head. Change the task and you change the head and retrain — the network still cannot take a free-form request and produce a free-form answer. What we actually want back is the base model's general text ability, but redirected: instead of continuing arbitrary web text, it should continue text of the form "here is a task; here is the answer." Nothing in the base model or the classifier teaches that behavior. The knowledge is there; the habit of answering is not.

Linear position

Forced by: a base model (11) only completes text and the classifier (12) does exactly one task — neither will take an arbitrary instruction and produce an arbitrary answer.
New idea: keep the whole language model and its cross-entropy loss (09), but train it on (instruction, response) examples wrapped in a fixed prompt template, and mask the loss to the response tokens so the gradient supervises the answer, not the copying of the question. That single change turns a completer into an assistant.
Forces next: this updates all ~124M parameters and stores a whole new copy of the model per task — expensive and duplicative. Lesson 14 learns a tiny low-rank patch instead (LoRA).

The plan

Five moves. (1) Pin down exactly why a base model has knowledge but not the behavior of answering. (2) Wrap every example in a fixed prompt template so the model learns the shape of a turn, and make train and inference use the same shape. (3) Assemble the data: (instruction, input, response) triples, batched with padding to the longest example and <|endoftext|> as the pad token. (4) Mask the loss to the response tokens — supervise the answer, ignore the prompt — reusing the same loss and loop. (5) Face the hard part: open-ended output is hard to score, so a stronger model becomes the judge. Then drive the template and the mask yourself.

1 · A base model has knowledge, not the behavior of answering

By lesson 11 you have a strong base model: its weights (loaded from GPT-2) put accurate probability on the next token across a huge slice of English. Ask it a question, though, and it does not answer — it continues. Type "List three primary colors." and a base model is just as likely to produce a fourth line of a list of instructions, or a paragraph about color theory, as it is to say "red, blue, yellow." It is doing precisely what it was trained to do: maximize p(next token | context) over generic text. Nothing in pretraining ever showed it that a request should be followed by a compliant response and then a stop.

So the gap is not knowledge — the model demonstrably knows the primary colors, because it can complete "The three primary colors are…" perfectly. The gap is behavior: the reflex to read a piece of text as an instruction addressed to it and to emit an answer in reply. That behavior is a statistical pattern like any other, which means we can install it the same way the model learned everything else — by showing it many examples of the pattern and letting cross-entropy do the work. We do not need a new loss, a new architecture, or a new head. We need new data, shaped so the pattern is unmistakable, and one adjustment to where the loss looks.

Why not just prompt it?

You can coax some answering out of a base model with a clever few-shot prompt ("Q: … A: …, Q: … A: …, Q: {yours} A:"). That works partially and wastes context on examples every single call. Instruction fine-tuning bakes the behavior into the weights: one short template at inference, no examples needed, and far more reliable formatting. Fine-tuning moves the demonstration from the prompt into the parameters.

2 · The prompt template — teaching the shape of a turn

The model reads only one thing: a stream of tokens. So "this part is the task, that part is your answer, and the answer ends here" must be written into the token stream. We do that with a fixed prompt template — a small, constant scaffold of text that wraps every example identically. The classic Alpaca-style template looks like this (an optional input field carries any data the instruction operates on, such as the sentence to translate):

Below is an instruction that describes a task. Write a response that appropriately completes the request. ### Instruction: {instruction} ### Input: {input} ### Response: {response}

Everything up to and including the ### Response: marker is the prompt; whatever follows it is the response. When an example has no input, the ### Input: block is simply dropped. The exact wording of the scaffold does not matter much — what matters is that it is fixed. The model learns to associate the marker tokens (### Instruction:, ### Response:) with "a task is being posed" and "now produce the answer." Those markers become a learned convention, exactly like any other token pattern.

The single most important rule follows from this: the template you train on must be the template you use at inference. At training the model sees the full triple, response included. At inference you feed it the template filled in up to ### Response: and let it generate the rest. If the two shapes differ even slightly — a different header, a missing newline, a renamed marker — the model is being asked to continue a pattern it never saw, and the crisp answering behavior degrades into base-model rambling. Consistency of the scaffold is what makes the learned reflex fire.

3 · The data — instruction/response triples, batched and padded

The training set is a list of examples, each a triple:

{
  "instruction": "Translate the sentence to French.",
  "input":       "The weather is nice today.",
  "response":    "Il fait beau aujourd'hui."
}

You do not need many. Instruction tuning is teaching a behavior, not new facts, so a modest, diverse, high-quality set goes a long way — the Alpaca dataset is on the order of tens of thousands of examples, and careful curations show that roughly a thousand strong, varied examples can already flip a base model into a serviceable assistant. Diversity of task types (summarize, classify, rewrite, extract, reason) matters more than raw count, because you are teaching the general habit of following instructions, not memorizing answers.

Batching: pad to the longest in the batch

To train efficiently we process many examples at once, stacked into a tensor of shape (B, T). But encoded examples have different lengths, and a rectangular tensor needs one common T. The fix is padding: pick a length for the batch and top up shorter sequences with a filler token. Rather than pad every batch to the global maximum (wasteful), pad each batch only to the longest example in that batch — the sequences you actually stacked together.

Which token is the filler? We reuse the special token the tokenizer already has for document boundaries: <|endoftext|>. It plays double duty — it is both the marker that ends a generated answer (so the model learns to stop) and the padding token that fills the tail of short sequences. Using one reserved token for both keeps the vocabulary clean and never collides with real content.

The trap

Padding tokens are filler, not signal. If you let the loss score the model on predicting pad tokens, you spend gradient teaching it to emit <|endoftext|> in the middle of nowhere, and the padded columns swamp the real ones in long-tail batches. Padding positions must be excluded from the loss — the same masking machinery we are about to build for the prompt handles them too.

4 · Mask the loss to the response

Here is the move that actually converts a completer into an assistant. The template gives every example two spans: the prompt (instruction + optional input + markers) and the response (the answer + its stop token). We want the model to learn to produce the response given the prompt. We do not want it to get better at reproducing the instruction — the user supplies that at inference; making the model good at echoing it is wasted, and worse, it dilutes the signal we care about with the signal we do not.

So we supervise only the response tokens. The loss is the same cross-entropy from lesson 09, evaluated at every position — but we replace the target id at every prompt position (and every pad position) with a sentinel that the loss function skips. In PyTorch that sentinel is the ignore_index of cross_entropy, conventionally -100:

IGNORE = -100                       # cross_entropy drops these targets

targets = input_ids[1:]             # next-token targets (shifted by one, from 03)
# mask the prompt: zero out supervision on instruction + input + markers
targets[: response_start - 1] = IGNORE
# mask the padding tail as well
targets[response_len_end:]    = IGNORE

loss = cross_entropy(logits.reshape(-1, V),
                     targets.reshape(-1),
                     ignore_index=IGNORE)   # averaged over UNMASKED positions only

Read what this does to the gradient. At a response position, the model is graded on whether it put probability on the correct next answer token — so it learns to answer. At a prompt or pad position, the target is -100, the term is dropped, and no gradient flows — so the model is neither rewarded nor punished for what it predicts there. The average is taken over the unmasked (response) positions only. Same loss, same optimizer, same training loop from lesson 10; the only new ingredient is where the loss is allowed to look.

Two masking subtleties

The stop token at the end of the response is supervised — that is how the model learns to end its answer instead of running on forever; do not accidentally mask it off. And the shift-by-one from lesson 03 means the boundary is response_start − 1: the target aligned with the last prompt token is the first response token, and we want that transition to be learned. Off-by-one here quietly trains the model to predict the prompt.

5 · Evaluating an assistant is hard

Classification (12) had a clean metric: accuracy against gold labels. Instruction following does not. There is no single correct summary, no unique polite rewrite — a good answer is one of many acceptable ones, and exact-match against a reference is far too strict. So how do you know fine-tuning worked, or which checkpoint is better?

The workable modern answer is to use a stronger model as a judge: feed a capable model the instruction and the candidate response and ask it to score helpfulness, correctness, and format, or to pick the better of two responses. It is imperfect — judges carry their own biases (length, style, position) — but it scales to thousands of prompts far more cheaply than human raters and correlates well enough to guide development. This is a whole subject of its own; the preference-and-evaluation machinery, including how judgments become a training signal (RLHF/DPO), lives in cs336 · 17 Preferences. For our purposes the takeaway is narrower: the template and the response-mask are what install the behavior; measuring how good that behavior is opens a new, harder problem.

6 · Format the triple and set the mask

The widget renders one (instruction, input, response) triple into the prompt template as token chips: dim chips are the prompt (instruction, input, and the scaffold markers), blue chips are the response. Toggle mask prompt tokens. With the mask on, only the response chips are outlined as supervised and the prompt chips fade to ignored — the model is graded solely on producing the answer, so it learns to answer. With the mask off, every non-pad chip is supervised, so the gradient also pushes the model to reproduce the instruction — it learns to parrot the question. The pad chips (from stacking with a longer example in the batch) are always ignored. The KPIs count supervised tokens versus the total and name the behavior each setting installs.

Instruction format + loss mask

Proves the two moves that turn a completer into an assistant: the fixed template gives the model the shape of a turn, and the response-only mask makes the gradient supervise the answer, not the copying of the question. Flip the mask off and watch the objective become "parrot the instruction."

Example:

prompt — instruction, input & markers response — the answer + stop token

supervised / total

—

supervised tokens

—

prompt tokens

—

learns to

—

The template rendered as tokens. Outlined = supervised; faded = ignored by the loss.

Failure modes & checklist

Failure modes

Train/inference template mismatch. Fine-tune with one scaffold, prompt with another. Signal: the model answers well in your eval harness but rambles in the product, because the marker tokens it learned are not the ones it is being fed.
Supervising the prompt. Forget the mask and the loss grades the model on reproducing the instruction. Signal: most of the gradient goes to echoing the question; answering is weak and format is mushy.
Masking the stop token. Exclude the response's <|endoftext|> from the loss and the model never learns to end — it answers, then keeps generating.
Scoring pad tokens. Leave padding in the loss and long-tail batches train the model to emit <|endoftext|> mid-sequence.
Off-by-one at the boundary. Mask up to response_start instead of response_start − 1 and you drop the crucial prompt→answer transition.

Checklist

One fixed template used at both train and inference; optional input block dropped cleanly when empty.
Data as (instruction, input, response) triples; diverse task types over raw volume.
Batches padded to the longest example in the batch with <|endoftext|>.
Targets set to -100 on prompt and pad positions; response (incl. stop token) supervised.
Same cross-entropy (09) and same AdamW loop (10) — only the mask is new.

Checkpoint

Try it

Pick the "translate (has input)" example and leave the mask ON: note the "supervised / total" fraction — only the short French answer and its stop token are graded. Now switch the mask OFF and watch the fraction jump as every instruction and marker chip lights up, and the "learns to" verdict flip from answer to parrot. Then switch to the "antonym (no input)" example: the ### Input: block vanishes entirely, yet the prompt/response split still holds. Ask yourself: of the three properties from lesson 00 — accurate, trainable, steerable — which one is the response-only mask protecting?

Where this points next

Instruction fine-tuning gives you a real assistant, but look at the cost. Every gradient step updated all ~124 million parameters, and the artifact you save is a complete new copy of the model — 124M numbers — dedicated to this one instruction-following behavior. Want a second specialization (a code assistant, a medical summarizer)? That is another full copy, another full training run over every weight. Most of that update turns out to be redundant: the change needed to adapt a pretrained model to a task is small and low-rank. Lesson 14, LoRA — parameter-efficient fine-tuning, freezes the base weights and learns a tiny low-rank patch instead, shrinking the per-task cost by orders of magnitude while keeping almost all of the benefit.

Takeaway

A base model has the knowledge but not the behavior of answering — it continues text, it does not respond. Instruction fine-tuning installs that behavior with the smallest possible change: keep the whole model and the identical cross-entropy loss, but train on (instruction, input, response) triples wrapped in a fixed prompt template, and mask the loss to the response tokens (targets set to -100 on the prompt and on <|endoftext|> padding). The template teaches the shape of a turn and must be identical at train and inference; the response-only mask makes the gradient supervise the answer instead of the copying of the question. Same loss (09), same loop (10) — only where the loss looks is new. Evaluating the result is the hard part, which hands off to model-as-judge and preference optimization. And because this touches every parameter and stores a full copy per task, it forces the cheaper adapter of lesson 14.

Interview prompts

Why doesn't a strong base model answer a question out of the box? (§1 — it is trained to maximize p(next token) over generic text, so it completes; it has the knowledge but never learned the behavior of reading text as an instruction and replying.)
What is the prompt template for and what is the one non-negotiable rule about it? (§2 — it writes "task here / answer here / ends here" into the token stream so the model learns the shape of a turn; it must be identical at training and inference or the behavior degrades.)
What does response-only loss masking do, and how is it implemented? (§4 — it supervises only the answer by setting prompt/pad targets to the ignore_index (−100) so cross_entropy drops them; same loss, gradient flows only on response tokens.)
Why is <|endoftext|> used as the padding token, and why must its response copy stay unmasked? (§3/§4 — it is a reserved special token that never collides with content and doubles as the stop marker; the response's stop token is supervised so the model learns to end, but pad copies are masked.)
What happens if you forget the mask and supervise the prompt too? (§4/failure modes — the gradient also trains the model to reproduce the instruction, diluting the answering signal and weakening format; most compute is spent echoing the question.)
How much data do you need, and what matters most about it? (§3 — not much; a diverse set of task types (~1k strong, varied examples can suffice) beats raw volume, because you are installing a behavior, not memorizing facts.)
Why is evaluating an instruction-tuned model harder than a classifier, and what is the usual workaround? (§5 — open-ended answers have no unique gold label, so exact-match fails; a stronger model is used as a judge to score or compare responses at scale → cs336/17.)

Companion reads: gpt_mini · 03 SFT shows the loss-mask as runnable code; cs336 · 16 SFT derives the chat template and masking in depth; cs336 · 17 Preferences covers judging responses and turning judgments into a training signal.