Part I — Text becomes numbers

The training signal — a sliding window

Lesson 02 turned each token id into a learnable vector and added a position signal, so we finally have a tensor the model can consume — a bag of input vectors. But a bag of inputs is not training data: gradient descent needs targets to push those vectors toward, and nobody labeled our text. This lesson shows the trick that makes the whole enterprise possible — the next token is the label — and turns any stream of text into a dense supply of (input → target) pairs, for free, with a sliding window.

What the last step left broken

After lesson 02 we can map a token-id sequence to an (B, T, d) tensor of embeddings — the exact input the transformer will eat. What we cannot do is learn anything from it. Training means "adjust the weights so a prediction gets closer to a known answer," and we have no known answers: our corpus is just raw text with no annotations attached. Buy a labeled dataset for the whole internet and the cost is astronomical; the model is dead in the water without something to predict.

Linear position

Forced by: lesson 02 gives input vectors but no labeled targets — training needs (input → target) pairs, and human annotation of a web-scale corpus is impossible.
New idea: self-supervision via a sliding window. For a token stream the target at each position is simply the next id — the input shifted left by one. Every token becomes a label; the supervision is free and dense. Choose a window width max_length and a stride; each window is an input row and its shift-by-one is the target row.
Forces next: to predict the target at position t, the model must flexibly combine the tokens at positions ≤ t — and a fixed recipe like "average them" is far too rigid, since which earlier tokens matter depends on the sentence. Lesson 04 must let each position weight the others by relevance: attention.

The plan

Four moves. (1) See why next-token prediction is a label that comes free with the text — self-supervision. (2) Slide a window of width max_length across the stream; each window is an input row, its one-step shift is the target row. (3) Tune reuse with stride: disjoint windows at one extreme, maximal overlap at the other; stack windows into a batch (B, T). (4) See why this is what lets LLMs be "large" — every token in a corpus is a training example. Then drive the sliding-window sampler yourself.

1 · The label is already in the text

Supervised learning needs pairs: an input and the correct answer for it. The whole difficulty of most machine learning is getting those answers — someone has to label the cats, transcribe the audio, mark the spam. For language modelling, that difficulty vanishes because of the objective we fixed back in lesson 00: predict the next token. The correct next token is not something we must go collect — it is sitting right there in the text, one position to the right.

Take a tokenized stream of ids and line it up against a copy of itself shifted left by one. The value under each input position is exactly the id that ought to come next:

input = [ id₀, id₁, id₂, id₃ ]
target = [ id₁, id₂, id₃, id₄ ] (the input, shifted by one)

Read it position by position and it becomes a stack of prediction tasks, each one honest: given [id₀] predict id₁; given [id₀, id₁] predict id₂; given [id₀, id₁, id₂] predict id₃; and so on. This is called self-supervision: the data supplies its own labels, so no human annotator is ever involved. It is the single fact that makes web-scale training feasible — the answer key is free because it is the corpus itself.

The input row

what the model sees

A contiguous slice of the token stream, positions p … p+L−1 for a window width L = max_length. This is what gets embedded (lesson 02) and fed forward.

The target row

what it must predict

The same slice moved one step right, positions p+1 … p+L. Aligned element-wise with the input, so position i of the input is asked to predict position i of the target.

Because the two rows are aligned, one forward pass over a length-L window produces L next-token predictions at once, and we get L supervised comparisons for the price of one — the model is trained at every position in the window simultaneously, not just at the end.

The trap: off-by-one

The target is the input shifted by exactly one: target[i] = input[i] + 1 position, never the same position and never two. Line them up wrong and you either train the model to copy its input (shift 0 — it learns the identity function and predicts nothing) or to skip a token (shift 2 — it learns a corrupted objective). When in doubt, check a single pair by hand: input [a, b, c] must pair with target [b, c, d].

2 · Slide a window across the stream

One window gives one input row and one target row. A whole corpus is far longer than any window the model can hold — GPT-2 caps its context at 1,024 tokens — so we chop the stream into windows and slide along. Two numbers control the chopping:

max_length — the context width L: how many tokens are in each input row (equivalently, how far back the model can look when predicting the last position). This is the T the transformer will see.
stride — the step: how far the window jumps to produce the next row.

Starting at position 0, take positions 0 … L−1 as input and 1 … L as target; jump forward by stride; take the next window; repeat until the window would run off the end of the stream. In pseudo-code the whole sampler is a single loop (paraphrased, not copied):

inputs, targets = [], []
for start in range(0, len(ids) - max_length, stride):
    inputs.append(  ids[start        : start + max_length    ] )   # positions p .. p+L-1
    targets.append( ids[start + 1     : start + max_length + 1] )   # shifted by one

Every row in inputs is one context; the aligned row in targets is its next-token answer key. Notice the loop stops at len(ids) - max_length: the last legal window needs one extra token on the right for its target, so we never step past where a full shifted row exists. (In practice, a partial final batch is simply dropped so every batch has the same shape.)

3 · Stride tunes overlap, and batching stacks the rows

The stride is the one real knob, and it trades coverage against reuse. Two settings anchor the range:

stride	windows	each token is supervised…	character
= max_length	disjoint — they tile the stream edge to edge with no gaps and no overlap	about once (as a target)	maximum coverage per pass, no duplication; the efficient default for large corpora
= 1	maximal overlap — the window inches forward one token at a time	up to L times, in L different contexts	squeezes the most pairs out of a small corpus, but rows are highly redundant
in between	partial overlap	between 1 and L	a dial between the two extremes

Intuitively: with stride = max_length the windows sit side by side like floor tiles, so almost every token appears in exactly one input row and is asked-about once. Drop the stride to 1 and consecutive windows overlap almost completely, so the same token is seen inside many different contexts — useful when data is scarce, wasteful (and a mild overfitting risk from near-duplicate rows) when it is plentiful. There is no gap and no double-counting only at the tiling setting; that is why stride = max_length is the natural choice once you have enough text.

Finally, a single window is one row of shape (T,). Training runs on batches: stack B windows and their B target rows to get two tensors of shape (B, T) — the inputs and the targets. Embedding lookup (lesson 02) then lifts the inputs to (B, T, d), the exact tensor the transformer consumes, while the targets stay as ids for the loss (lesson 09) to compare against. Larger batches give steadier gradient estimates at the cost of memory — a hyperparameter, not a correctness issue.

4 · Why this is what lets LLMs be "large"

Count the pairs. A stream of N tokens sampled with stride = 1 yields roughly N input rows, and each row of width L carries L aligned next-token predictions inside it — so a single short story is already tens of thousands of supervised examples, and it cost nothing to label. Scale that up: a book is millions of pairs, and the readable internet is essentially all of them.

This is the quiet engine behind the word "large" in large language model. The bottleneck for supervised learning is almost always labeled data; here the labels are free, generated mechanically from the text, so the amount of training signal is limited only by how much text exists — which is effectively unbounded. There is no annotation budget to blow, so the data can grow to match ever-larger models. Every emergent ability we admire is paid for by this: dense, self-generated supervision at a scale no human labeling effort could ever reach.

The trap: windows across document boundaries

A raw corpus is many independent documents concatenated. If a window straddles the seam between two of them, the model is trained to "predict" the first token of document B from the tail of document A — a spurious dependency that never should exist. The standard fix is to insert an <|endoftext|> token between documents; the model learns that this marker resets context, so it stops carrying meaning across the boundary. (This is the same special token introduced in lesson 01.)

5 · Slide the window yourself

The widget below holds a fixed stream of twenty tokens (short words, shown as colored chips with their stream position underneath). The two sliders are the sampler's knobs: max_length sets the window width and stride sets the jump. It draws two consecutive windows so the overlap (or the lack of it) is visible at a glance — each window as an input row and, directly below it, the target row shifted one chip to the right (outlined). Press Next window to advance the starting position and watch the window march along the stream. The KPIs report how many (input, target) pairs the whole stream yields at these settings, how many tokens are reused because of overlap, and how many times an average token gets supervised — the numbers that make "free and dense" concrete.

Failure modes & checklist

Failure modes

Off-by-one targets. Shift 0 trains the identity (copy the input); shift 2 skips a token. Signal: loss drops implausibly fast (it's learning to copy) or never learns a sensible objective. The target must be the input shifted by exactly one.
Windows crossing document boundaries. Predicting doc B's first token from doc A's tail injects a fake dependency. Fix: separate documents with <|endoftext|>.
stride too small on a big corpus. Near-duplicate rows waste compute and can mildly overfit. Prefer stride = max_length when data is plentiful.
Running off the end. The last window needs a token to its right for the target; stop the loop at len(ids) − max_length (and drop the ragged final batch).

Checklist

Target row = input row shifted left by one (target[i] = input[i+1]).
Input is positions p … p+L−1; target is p+1 … p+L.
Loop stride advances the start; loop bound is len(ids) − max_length.
stride = max_length → disjoint tiles; stride = 1 → full overlap.
Batch = stack B rows → inputs and targets both (B, T).
<|endoftext|> between documents so context resets at seams.

Checkpoint

Try it

Set max_length = 4 and stride = 4. Read the two windows: they sit edge to edge with no shared chips, and the "tokens reused" KPI reads 0 — this is the disjoint tiling that supervises each token about once. Now drag stride down to 1 and watch the second window slide back so it overlaps the first by three chips; "each token supervised" climbs toward L, and "pairs from stream" jumps. You have just dialed the same knob the data loader exposes. Which of the three properties from lesson 00 — accurate, trainable, steerable — does a free, dense training signal make possible in the first place?

Where this points next

We now have honest training data: a stream of (input, target) pairs, as many as we want, at no labeling cost. But look closely at what a single pair actually demands. To predict the target at the last position, the model has to combine everything in the input window — and the way it should combine them is not fixed. In "the cat that chased the mouse was tired," the word that resolves "was" is "cat," seven tokens back, not the adjacent "mouse." A rigid recipe like "average the window" or "just use the previous token" cannot express that; the right earlier tokens depend entirely on the sentence. What we need is a mechanism that lets each position weight every earlier position by how relevant it is, and learn those weights from exactly the pairs we just built. That mechanism is attention — lesson 04, Attention, the idea — weighting by relevance.

Takeaway

Training needs (input → target) pairs, and language supplies them for free: the target is the input shifted by one, so the next token is the label — self-supervision. A sliding window of width max_length chops the token stream into input rows, each paired with its one-step shift; stride tunes overlap — stride = max_length tiles the stream disjointly (each token supervised ~once), stride = 1 overlaps maximally (each token supervised up to L times). Stack B windows into (B, T) tensors and the pairs are ready for the loss. Because every token in a corpus is a training example, the supply of labels is bounded only by how much text exists — which is why LLMs can be trained on the whole internet, and why they can be "large."

Interview prompts

What is the label for next-token prediction, and why is it "free"? (§1 — the label at each position is the next token, already present in the text; the corpus is its own answer key, so no human annotation is needed — self-supervision.)
Exactly how are the input and target rows related? (§1 — the target is the input shifted left by one: target[i] = input[i+1]; aligned element-wise so one window yields L predictions.)
What do max_length and stride control? (§2/§3 — max_length is the context width per row; stride is how far the window jumps, which sets the overlap between consecutive windows.)
Contrast stride = max_length with stride = 1. (§3 — the former tiles the stream disjointly so each token is supervised ~once with no duplication; the latter overlaps maximally so each token is reused in up to L contexts, good for small corpora but redundant.)
Why does this scheme let language models be "large"? (§4 — every token is a training example and labels are generated mechanically, so the training signal is limited only by available text, which is effectively unbounded — no annotation budget caps the data.)
What goes wrong at document boundaries, and what fixes it? (§4 — a window straddling two documents trains a spurious cross-document dependency; inserting <|endoftext|> between documents makes the model reset context at the seam.)
Name the most common off-by-one bug in this pipeline. (§1 — pairing the target at the same position (shift 0) trains the identity/copy function; the target must be shifted by exactly one.)

Companion reads: cs336 · 13 Data takes this same sliding-window idea to the scale of the real pretraining pipeline — deduplication, filtering, packing, and streaming terabytes of text.