Part IV - Customization and training jobs

Model Customization Choices

Lesson 12 closed the inference story: you can serve a base model behind safety filters, quality checks, and guardrails. The natural next question from product teams is "make the model ours" — answer in our voice, know our docs, follow our policies. That request hides a trap: it sounds like "fine-tune a model," but fine-tuning is the most expensive way to satisfy most versions of it, and it drags a whole training-and-rollback apparatus onto your platform. This lesson is the decision you make before you provision a single GPU for training: pick the cheapest mechanism on a ladder of customization options that actually solves the product problem.

Source coverage

PDF Chapter 6: model customization, prompt engineering, fine-tuning, PEFT, and LoRA. This lesson reframes that material as an operational decision and connects it to the serving machinery from earlier lessons.

Linear position

Prerequisite: Lesson 12 (Safety, Quality, and Guardrails) gave you a serving stack that can run and police a base model. Lesson 09's LoRA-aware routing and vLLM multi-LoRA serving showed that one base model can host many adapters at once.
New capability: A decision rule that maps a product requirement to the lowest rung on the customization ladder that satisfies it — so you only take on training infrastructure (lesson 14) when nothing cheaper works.

The plan

Five moves. (1) Frame customization as a ladder — prompt, retrieval, adapter, full fine-tune, pretrain — where each rung buys more control and costs more machinery. (2) Define every rung precisely: what it changes, what it leaves frozen, and what knowledge or behavior it is the right tool for. (3) Make the choice mechanical with a decision rule and a cost/control comparison table. (4) Work the LoRA-vs-full-checkpoint numbers concretely — megabytes vs gigabytes — and show why that size gap is what lets one base model serve dozens of adapters. (5) Failure modes, a checklist, and the hand-off into lesson 14, which builds the actual fine-tuning jobs once you have decided you need one.

1 · Customization is a ladder, not a switch

When someone says "customize the model," they are almost never asking for one specific technique — they are describing an outcome ("it should know our return policy," "it should write in our brand voice," "it should stop hallucinating product SKUs"). Different outcomes are best served by different mechanisms, and those mechanisms form a ladder ordered by cost and depth:

prompt  ->  retrieval (RAG)  ->  adapter (LoRA/PEFT)  ->  full fine-tune  ->  pretrain
change the     inject knowledge      train small frozen-     update all          train a model
input only     at query time         base adapter weights    weights             from scratch
cheap / instant rollback  ----------------------------------------------------->  expensive / slow rollback

The mental model: climb only as high as the problem forces you to. Every rung above "prompt" introduces machinery you now own forever — data curation, a training job, an evaluation harness, a model/adapter registry, a deployment path, and a rollback plan. A prompt change is a one-line edit you can revert in seconds. A full fine-tune is a multi-gigabyte artifact you must version, A/B test, monitor for regressions, and be able to roll back to the previous checkpoint under incident pressure. The rungs are not a maturity ranking — shipping a fine-tuned model is not "more advanced" than shipping a good prompt; it is just harder to inspect and more expensive to be wrong about.

The trap: customization is a verb, fine-tuning is a noun

Product asks for an outcome ("our voice"); engineers hear an artifact ("a fine-tuned model"). The artifact is the most expensive rung that could deliver the outcome, not the cheapest one that will. The discipline of this lesson is to always translate the outcome back into the lowest rung first, and to make every escalation up the ladder a deliberate, evidence-backed decision rather than a default.

2 · The five rungs, defined

Define each rung by three things: what it changes, what it freezes, and the kind of problem it is the right answer to.

Prompt / context engineering

Changes the input, trains nothing. You edit the system prompt, add few-shot examples, or restructure the context the model sees. Weights are untouched. Instant to change, instant to roll back, $0 training cost. Right when the knowledge or behavior you want is already latent in the base model and just needs to be elicited.

RAG (retrieval-augmented generation)

Injects retrieved knowledge at query time. A retriever fetches relevant documents and the system splices them into the prompt before generation; weights are frozen. Right when the knowledge changes (prices, inventory, policies) or when you need citations / provenance — you can point at the source document.

PEFT / LoRA

PEFT = Parameter-Efficient Fine-Tuning: train a tiny set of new parameters and freeze the rest. LoRA = Low-Rank Adaptation, the dominant PEFT method: train small low-rank adapter matrices that add onto the frozen base weights. Cheap to train, small to store, swappable, and serveable many-at-once off one base. Right when behavior or style must shift and you have labeled examples.

Full fine-tuning

Updates all model weights. Produces a complete new multi-GB checkpoint. Most control, most cost: large GPU jobs, a full deployment-and-rollback story per version, and no free multi-tenant sharing. Right only when adapter capacity is genuinely insufficient or serving constraints demand merged weights.

Pretrain from scratch

Trains a new base model. Months of compute, a data pipeline, and a research team. Off the table for almost every product problem; included only to mark the top of the ladder. If you are considering this to "add knowledge," you are on the wrong rung by four steps.

The distinction that trips people up most is RAG vs fine-tuning. They feel like competitors but answer different questions. RAG changes what the model knows at this moment by handing it documents; fine-tuning changes how the model behaves in general by adjusting weights. Knowledge that changes weekly belongs in retrieval, where updating it is re-indexing a document — not retraining a model. Behavior that should be consistent regardless of input (tone, format, domain reasoning style) belongs in weights.

3 · The decision rule and the cost/control table

The choice can be made almost mechanically. Walk down the rungs and stop at the first one that satisfies the requirement:

→knowledge already in the base model → prompt / context engineering. Just elicit it.

→knowledge changes, or citations/provenance matter → RAG. Put facts where they can be re-indexed and cited.

→behavior / domain style must change and you have examples → LoRA / PEFT. Train a small adapter.

→adapter capacity or serving constraints justify the cost → full fine-tune. Only here do you own a new checkpoint.

The cost/control table makes the escalation tax explicit. "Ops cost" is the standing machinery you take on, not just the one-time training bill:

Rung	What it changes	Data needed	Ops cost	Rollback
Prompt / context	The input only; weights frozen	None (a few examples)	None — a config edit	Instant (revert the prompt)
RAG	Knowledge injected at query time; weights frozen	A document corpus + index	Retriever, vector index, freshness pipeline	Fast (re-index or disable retrieval)
LoRA / PEFT	Small low-rank adapter; base frozen	Hundreds–thousands of labeled examples	Training job, eval, adapter registry, multi-LoRA serving	Fast (swap/unload the adapter — base unchanged)
Full fine-tune	All weights → new checkpoint	Large curated dataset	Large GPU job, registry, per-version deploy + monitoring	Slow (redeploy the previous multi-GB checkpoint)
Pretrain	A new base model from scratch	Web-scale corpus	A research program	N/A — you would not

Rules of thumb that fall out of the table

Do not train facts into weights if those facts change — that is a retrieval problem wearing a training costume. Do not build a RAG pipeline for behavior that should simply be learned once (brand voice is not a document lookup). Do not fine-tune until a stronger prompt and a RAG baseline have been measured and shown insufficient. A fine-tuned model with no eval gate is not more mature than a prompt — it is just a config change you can no longer read.

4 · The number that matters: adapter MB vs checkpoint GB

The reason LoRA sits so much lower on the ladder than full fine-tuning is not just training cost — it is artifact size, and that size determines how you serve it. Work a concrete example with a typical 7-billion-parameter model.

A full fine-tune updates every weight, so the output is a complete new checkpoint. At 7B parameters in 16-bit precision that is 7 × 10⁹ × 2 bytes ≈ 14 GB per version. Every fine-tune you ship is another 14 GB to store, load into GPU HBM, version, and roll back.

A LoRA adapter trains only small low-rank matrices added alongside the frozen base. With rank r = 16 applied to the attention projections, the adapter typically lands in the single-digit-to-low-hundreds-of-MB range — call it ~40 MB for this model. That is roughly 14,000 MB / 40 MB ≈ 350× smaller than the full checkpoint.

Why the size gap changes serving, not just storage

Because the base weights are shared and frozen, you load the 14 GB base into GPU HBM once and then hot-load many tiny adapters on top of it. Purely by storage, the free HBM around a 14 GB base on a 40 GB card could hold hundreds of ~40 MB adapters; in practice a vLLM server running multi-LoRA caps the resident set far lower — typically dozens active at once (the max_loras / max_cpu_loras knobs) because each active adapter adds scheduling and memory-traffic overhead, not just bytes. Either way it routes each request to its tenant's adapter — one base model, many customized behaviors, one set of GPUs. A full fine-tune cannot do this: each 14 GB checkpoint is its own model needing its own HBM and (often) its own replica. So "use LoRA" is never only a training decision — it is the thing that makes per-tenant or per-task customization affordable to serve (this is exactly the LoRA-aware routing of lesson 09).

The interactive below lets you climb the ladder and watch control, machinery, and serving model trade off rung by rung.

The customization ladder — control vs machinery vs serving footprint

Pick a rung. The bars show control (how deeply you can reshape the model) against ops machinery (the standing infrastructure you take on). The KPIs show the artifact size and how many customized behaviors share one 14 GB base in HBM. Note the gap between the storage ceiling (free HBM ÷ adapter size — hundreds) and what you actually keep active at once (dozens): a multi-LoRA runtime caps resident adapters well below the storage limit because each active adapter adds scheduling and memory-traffic overhead, not just bytes. Climb only until control crosses the line the problem requires — then stop, because machinery keeps rising.

rung: prompt

Artifact size

—

Adapters per 14 GB base

—

Rollback

—

Right when

—

Show the core JS

// each rung carries (control, machinery, artifactMB).
// "behaviors per base" = how many customizations share one 14 GB base in HBM:
//   prompt/RAG/LoRA share the SAME frozen base -> many; full fine-tune does not -> 1.
const RUNGS = [
  {name:"prompt",  control:1, machinery:0, mb:0},      // input only
  {name:"RAG",     control:2, machinery:2, mb:0},      // weights frozen
  {name:"LoRA",    control:3, machinery:3, mb:40},     // ~40 MB adapter, base shared
  {name:"full ft", control:4, machinery:5, mb:14000},  // 14 GB, own replica
  {name:"pretrain",control:5, machinery:6, mb:14000},
];
// fanout: shared-base rungs hold ~ (HBM_free / adapterMB) adapters; full ft = 1.

5 · Where customization goes wrong

Failure modes

Fine-tuning facts that should live in retrieval. A team bakes the current product catalog or pricing into weights; a week later the facts are stale and the only fix is another training run. Signal: your retraining cadence tracks how often data changes rather than how often behavior needs to change — that knowledge belonged in RAG.
Shipping a customized model with no baseline comparison. The fine-tune ships because it "feels better," never measured against a strong prompt or a RAG setup. Signal: there is no eval table showing the fine-tune beating the cheaper rungs — so you cannot tell whether the training was necessary or just expensive.
Treating LoRA as a training-only decision. An adapter is trained in isolation, then someone discovers serving it means standing up multi-LoRA routing, an adapter registry, and HBM budgeting that nobody planned. Signal: the training job is "done" but the model still cannot reach a request, because the serving path for adapters was never designed (lesson 09).
Climbing past the rung the problem needed. A prompt would have solved a tone request, but the team fine-tuned anyway — now they own a checkpoint, a rollback plan, and a monitoring burden for a one-line change. Signal: ops cost rose sharply while the measured quality gain over the prompt is within noise.
No rollback story for the chosen rung. A full fine-tune regresses a key behavior in production and there is no fast path back to the previous checkpoint. Signal: an incident where "revert" means "retrain," because the previous artifact was not retained and versioned.

Implementation checklist

Did you start at prompt / context when the needed knowledge already exists in the base model or your data sources, and measure it before climbing?
Is changing or freshness-sensitive knowledge — plus anything needing citations or provenance — served by RAG rather than baked into weights?
Are you using LoRA / PEFT (not full fine-tuning) when only behavior or style must change and labeled examples exist?
Is full fine-tuning reserved for the case where adapter capacity is genuinely insufficient or serving constraints demand merged weights — and is that justified with evals?
For any adapter, have you planned the serving path — multi-LoRA routing, adapter registry, HBM budget — not just the training job?
Does each rung you ship have an explicit, fast rollback (revert prompt / re-index / unload adapter / redeploy previous checkpoint)?
Is there an eval table where the chosen rung beats every cheaper rung it skipped?

Checkpoint exercise

Try it

Take three real product requests — "answer in our brand voice," "always cite the source policy document," and "stop confusing our two SKU formats" — and assign each to a rung using the §3 decision rule. For each, name the data you would need, the standing ops cost you would take on, and the rollback path. Then pick the one request where someone might reflexively reach for full fine-tuning, and write the one-sentence argument for why a cheaper rung is the right first attempt.

Where this points next

This lesson is the decision; lesson 14, Fine-Tuning Jobs on Kubernetes, is the build. Once the ladder has told you that a prompt, RAG, and even a LoRA adapter are insufficient — or that you need to train that adapter in the first place — you have to actually run the training workload on the cluster: requesting multi-GPU gang-scheduled resources, mounting datasets, checkpointing to survive preemption, and writing the resulting artifact to a registry your serving stack can pull. Lesson 14 turns "we decided to train" into a concrete Kubernetes Job with the GPU, storage, and fault-tolerance machinery that decision now requires.

Takeaway

Customization is a ladder — prompt, RAG, LoRA/PEFT, full fine-tune, pretrain — and each rung buys more control while taking on more data, evaluation, deployment, and rollback machinery that you own forever. Use the §3 decision rule: knowledge already latent → prompt/context; knowledge that changes or needs citations → RAG; behavior/style with examples → LoRA/PEFT; only when adapters cannot express the change or serving demands merged weights → full fine-tune. The deciding number is artifact size: a ~40 MB LoRA adapter is ~350× smaller than a 14 GB full checkpoint, and because the frozen base is shared, one base model in HBM serves dozens of adapters at once — so LoRA is a serving decision as much as a training one. Climb only as high as the problem forces, measure each rung against the cheaper one below it, and keep a fast rollback for whatever rung you ship.

Interview prompts

Why is "customize the model" not the same request as "fine-tune the model"? (§1 — customization is an outcome that sits on a ladder of mechanisms; fine-tuning is the most expensive rung that could deliver it, not the cheapest one that will. The discipline is to translate the outcome back to the lowest rung first.)
RAG vs fine-tuning — which knowledge goes where? (§2 — RAG changes what the model knows right now by handing it documents and supports citations; fine-tuning changes general behavior via weights. Knowledge that changes or needs provenance belongs in retrieval; consistent behavior/style belongs in weights.)
Define PEFT and LoRA and say what each freezes. (§2 — PEFT trains a tiny new parameter set and freezes the rest; LoRA is the dominant PEFT method that trains small low-rank matrices added onto the frozen base weights.)
State the decision rule for picking a customization rung. (§3 — knowledge already in the model → prompt; changes or needs citations → RAG; behavior/style with examples → LoRA/PEFT; adapter capacity or serving constraints justify it → full fine-tune. Stop at the first rung that satisfies the requirement.)
Why is LoRA a serving decision, not just a training one? (§4 — a ~40 MB adapter on a shared frozen base lets one 14 GB base in HBM host dozens of adapters via multi-LoRA routing; a full 14 GB checkpoint needs its own HBM/replica. Forgetting the serving path is a classic failure (§5).)
Work the size comparison between a LoRA adapter and a full fine-tune of a 7B model. (§4 — full checkpoint ≈ 7×10⁹ × 2 bytes ≈ 14 GB per version; a rank-16 adapter ≈ 40 MB, roughly 350× smaller, which is what enables many-adapters-per-base serving.)
Someone wants to fine-tune to add this week's product catalog. What do you tell them? (§3, §5 — that is changing knowledge, so it belongs in RAG where updating is re-indexing, not retraining; fine-tuning facts that change is a top failure mode whose signal is retraining cadence tracking data churn.)