The journey of one record

The orientation gave you the map — bronze, silver, gold, three regimes. Before we dissect each stage, let's walk a single, concrete record across that whole map exactly once, so the flow lands as one continuous story rather than eleven disconnected lessons.

Where we are

Orientation handed you the map: the medallion shape (bronze → silver → gold) and the three regimes (SFT, preference, RL). This interlude introduces no new machinery. It picks one data record and follows it — the same datum, traceable at every hop — from the moment a human types it to the moment the trainer reads it. Then Part I onward zooms into each stage in detail. Think of this as the trailer before the chapters.

Intuition — one parcel through a postal system

Picture a single parcel moving through a postal network, and follow only that one parcel the whole way.

It enters at a drop-box — someone hands it over. (the source)
At the receiving depot it gets a tracking label: when it arrived, who sent it, what's declared inside, a tamper-proof seal. The parcel itself is untouched — only the label is added. (bronze + provenance)
At a regional hub it's inspected, repacked into a standard box, weighed, and checked against everything already in the system so the same parcel isn't shipped twice. (silver: clean · dedup · check)
It's loaded into a shipping container alongside other parcels headed the same way — packed tight so the truck runs full, with dividers so one parcel's contents don't spill into another's. (gold: tokenize + pack)
It's delivered to its destination and opened. (the trainer)

You could put your finger on that one parcel at every step and say "that's still mine." That traceability — the same physical thing, accounted for at each hop — is exactly what we'll now do with one data record. Watch the median-of-a-list example below; it is our parcel.

Mechanics — the spine, with one record's state annotated

Here is the medallion spine from orientation, but annotated with the state of our one record at each hop. Read it top to bottom; every subsection below expands one column.

  SOURCE          BRONZE              SILVER                  GOLD                   TRAINER
  ──────          ──────              ──────                  ────                   ───────
  annotator   ─▶  raw + provenance ─▶ normalized · deduped ─▶ token_ids + mask  ─▶  one row in a batch
  writes          immutable           schema-coerced          packed w/ others       model reads it once
  (prompt,        (lesson 03)         (lessons 05–06)         (lesson 07)            & takes one step
   response)
  ............................. OUR RECORD ........................................................
  prompt:         record_id=          prompt/response         ids=[128000,40,...]    row r in batch B:
  "…median        sha256:c7d3…        unchanged in meaning,   mask=0 over prompt,    (token_ids, mask)
   of a list."    payload untouched   whitespace/unicode      mask=1 over response   contributes loss
  response:       license, ts,        normalized; survives    span; then packed      only on the
  def median(…)   pii_flag stamped    dedup (no twin)         into one length-L seq  response tokens

The record never changes identity — it is the same (prompt, response) pair the annotator wrote. What changes is its representation (text → tokens) and its packaging (a standalone row → one segment inside a packed sequence). The content hash stamped at bronze is the through-line: it rides along as a provenance column at every stage — even after the text becomes tokens packed beside other records — so you can always point back at this exact datum.

The record we'll follow

Our protagonist is a real human-annotation example — the same one that appears in lesson 03's bronze-record inspector. A human annotator was given a coding prompt and wrote a reference answer:

prompt:    "Write a Python function that finds the median of a list."

response:  def median(lst):
               s = sorted(lst)
               n = len(s)
               return s[n//2] if n%2 else (s[n//2-1]+s[n//2])/2

That pair — (prompt, response) — is the SFT unit of data from the orientation table. Everything below happens to this one pair. Keep your eye on it.

SOURCE — the annotator writes it

A human in a labeling vendor's queue is shown the coding prompt and writes the def median(...) answer. At this instant the record is just a raw human-annotation event: a prompt, a response, and whatever context the labeling tool emits (annotator pool, labeling-spec version, an agreement score). It is not yet stored anywhere durable, not yet trusted, not yet in any schema we control. It is the most upstream form our parcel will ever take — handed over at the drop-box.

Regime check

This is the SFT regime (supervised fine-tuning): a demonstration of the desired output for a given input. Lesson 01 contrasts this with the preference unit (prompt, chosen, rejected) and the RL unit (prompt, verifier) — and the coda below follows the RL cousin of this very record.

BRONZE — it lands immutably, with a provenance wrapper

The ingestion job lands the event in the bronze layer. Bronze is raw, immutable, and append-only: the payload is written exactly as received and never edited in place. The one thing the ingest job adds is a provenance wrapper — the tracking label on the parcel — captured at the door, because (as lesson 03 stresses) provenance can never be reconstructed after the fact.

{
  "record_id":    "sha256:c7d3e1f2a894b5...",   # content hash = identity + dedup key
  "source_type":  "human_annotation",
  "source_id":    "vendor=scale/project=coding-sft/batch=2024-11-04",
  "origin_url":   "s3://bronze/annotation/scale/2024-11-04/batch_007.jsonl.gz",
  "license":      "proprietary — Scale MSA §4.2",
  "consent_flag": true,
  "pii_flag":     false,
  "spec_version": "coding-sft-v3.1",
  "ingest_ts":    "2024-11-04T18:22:01Z",        # wall-clock at landing
  "pipeline_run": "ingest-annotation-20241104-1822",
  "payload": {                                    # ← untouched, exactly as written
    "prompt":   "Write a Python function that finds the median of a list.",
    "response": "def median(lst):\n    s = sorted(lst)\n    n = len(s)\n    return s[n//2] if n%2 else (s[n//2-1]+s[n//2])/2",
    "regime":   "sft",
    "quality":  { "annotator_agreement": 0.94 }
  }
}

Note the record_id: a SHA-256 content hash over the payload bytes. That hash is our parcel's tracking number — it pins this exact datum's identity for the rest of the journey, and doubles as the key that prevents the same bytes from landing twice. From here on, "our record" means "the record with record_id sha256:c7d3…".

This hop is owned by lesson 03

Ingestion & provenance — the five source archetypes, append-only landing, content-hash dedup at the door, and why every provenance field must be stamped at ingest — is dissected in lesson 03. Here we only need: the record lands raw, immutable, and labeled.

SILVER — normalized, checked, deduplicated, schema-coerced

The silver transform is the regional hub: it turns the raw payload into a clean, canonical unit. Our record goes through four checks, and survives all of them:

Normalization. Whitespace and Unicode are canonicalized — trailing spaces trimmed, Unicode NFC-normalized, line endings unified. Our code's meaning is unchanged; the bytes are now consistent with every other record's.
PII check. The content is scanned for personal data. The prompt is a generic coding question and the response is plain code — pii_flag stays false, so the record passes untouched. (A record that did contain PII would be redacted or dropped here.)
Content-hash dedup. The record's hash is checked against everything already in silver. There is no twin — this exact (prompt, response) hasn't been seen — so it survives. Had an identical answer landed from another vendor batch, one copy would be dropped here.
Schema coercion. The loose payload is coerced into the canonical SFT unit: a typed (prompt, response) record with the regime tag. This is the unit every downstream stage expects.

What would have killed our parcel at this hub: a near-duplicate (a trivially reworded copy of an answer already kept — caught by fuzzy dedup, not exact-hash), or a failed quality gate (e.g. an empty response, a response that doesn't parse, or an annotator-agreement score below threshold). Ours is clean, distinct, and well-formed, so it walks straight through.

This hop is owned by lessons 05–06

The silver transform — distributed normalization and cleaning (lesson 05), and exact + near-duplicate dedup plus eval-set decontamination (lesson 06) — is two full lessons. Quality gates get their own treatment in lesson 08. Here we only need: the record comes out clean, canonical, and confirmed unique.

GOLD — tokenized to integers, masked, and packed

The gold transform converts the clean text into exactly what the model consumes: integer token_ids plus a loss_mask, then packs it with other examples. This is the shipping-container step.

Tokenize. A pinned tokenizer chops the text into tokens and maps each to an integer ID. Our record becomes two integer spans concatenated — the prompt's ids followed by the response's ids:

  text  →  "Write a Python function … median of a list."   def median(lst): … /2
           └──────────── prompt tokens ───────────────┘   └─── response tokens ───┘

  token_ids = [ 128000, 8144, 264, 13325, 734, …,  1759, 13 ,   711, 14288, 76, …, 17 ]
                └──────────── prompt span ─────────────┘   └────── response span ──────┘
  loss_mask = [   0,     0,    0,    0,    0,  …,   0,    0 ,    1,    1,    1,  …,  1  ]
                └──────────── mask = 0 ────────────────┘   └────── mask = 1 ───────────┘
                       (we do NOT grade the prompt)              (we DO grade the answer)

Loss mask. The mask is 1 only over the response span (the def median… code) and 0 over the prompt. The one-line reason: we grade only what the model should learn to generate. The prompt is given context — the model didn't produce it and shouldn't be rewarded or penalized for predicting it — so it contributes no loss. Only the answer tokens carry gradient signal. (Lesson 01 encodes the same instruction the way the trainer sees it — masked positions get the label -100, which cross-entropy skips; here mask=0 is that same "don't grade this." This per-regime masking rule is lesson 01's data unit made concrete, and lesson 07 derives it from the chat-template structure.)

Pack. Our record is short. Rather than pad it out to the full context length L and waste the rest, the packer concatenates it end-to-end with other short examples into one fixed-length sequence — and inserts a block-diagonal boundary so attention can't cross between documents (and position IDs reset at each boundary):

  one packed sequence (length L):

  [ … doc_A … │ OUR RECORD: prompt+response │ … doc_C … │ pad ]
              ▲                              ▲
              └── block-diagonal boundary ───┘
                  our record attends ONLY to itself —
                  it can't peek at doc_A or doc_C

Our parcel is now one labeled segment inside a shared container — packed tight so the truck (the GPU) runs full, with dividers (the block-diagonal mask) so its contents stay separate from its neighbors'. Its loss_mask still marks only its own response span.

This hop is owned by lesson 07

Tokenization & packing — tokenizer determinism, per-regime loss masking, first-fit-decreasing bin-packing, and the block-diagonal attention mask that makes packing correct — is lesson 07. Here we only need: text became token_ids + a loss_mask, and got packed into one fixed-length sequence.

TRAINER — it arrives as one row and is read once

Finally the packed sequence reaches the trainer as one row of (token_ids, loss_mask) inside a batch. The model reads it, computes the loss only over our record's response tokens (where the mask is 1), and takes one gradient step. Our parcel has been delivered and opened. In a static SFT run it is typically read once per epoch and then the next batch arrives — the journey, for this record, is complete.

The RL cousin

The same end-to-end idea, one regime over. In RL the record is not (prompt, response) but (prompt, verifier) — there is no human-written response. Imagine our example shipped as a prompt plus a checker: "Write a Python function that finds the median of a list" paired with a verifier that runs the function against test cases and scores it. The response is generated by the model itself during training, scored by the verifier, and the scored attempt is fed straight back as the training signal — then the loop repeats with the now-updated policy. Instead of a one-way batch trip from source to trainer, the record makes one lap around an online loop:

  prompt + verifier ─▶ model generates response ─▶ verifier scores it ─▶ trainer step ─┐
        ▲                                                                              │
        └──────────────── policy updated; loop again (one trip per step) ──────────────┘

So the batch journey is a straight line you walk once (source → bronze → silver → gold → trainer); the RL journey is that same pipeline bent into a circle that runs every step, with the response manufactured live rather than landed from a source. Lesson 10 dissects this online dataplane — replay buffers, staleness, backpressure — but the through-line is identical: a unit of data flows through ingest, clean, gate, and into the trainer.

Takeaway

You've now seen the whole flow once, concretely — the same record, traceable at every hop by its content-hash id (which travels alongside it as a provenance column even once the text becomes packed tokens): written by an annotator, landed raw with provenance in bronze, cleaned and confirmed unique in silver, turned into masked token_ids and packed in gold, then read once by the trainer (or, in its RL cousin, looped live around a verifier). Each lesson ahead zooms into one hop of this exact journey: lesson 03 owns bronze, 05–06 own silver, 07 owns gold, and 10 owns the RL loop. Next, Part I starts at the beginning — the data itself, by regime, before any pipe touches it.