Part I — Text becomes numbers

Embeddings & positions — ids become meaning

Lesson 01 turned text into a sequence of integer ids with byte-pair encoding, so the function finally has numbers to eat. But those numbers are just names on a coat-hook: id 5 is no more "like" id 6 than it is like id 5000, and a network that multiplies them will read that ordering as if it were meaning. This lesson replaces each id with a learnable vector, then adds a second vector that records where the token sits — producing the exact tensor the transformer blocks consume.

What the last step left broken

An integer id is an arbitrary label. The BPE tokenizer might assign "cat" the id 4127 and "dog" the id 91, purely by where they landed during the merge process — the two ids carry no hint that both are animals. Yet the very first thing a neural net does is multiply its input by a weight and add. If we feed the raw id in, the model literally sees "4127" as a number forty-five times larger than "91," and it will (wrongly) treat that magnitude, and the accidental ordering of the whole vocabulary, as structure to exploit. We need a representation where distance means similarity and every id starts on equal footing.

Linear position

Forced by: tokenization (01) leaves us with integer ids whose magnitude and ordering are arbitrary — a net would misread id-size as meaning, and there is no way for "similar tokens" to be "nearby."
New idea: look each id up in a learnable (V, d) embedding table — row i is token i's d-dimensional vector, trained by backprop so semantically related tokens drift together; then, because attention is order-blind, add a second (context_length, d) positional table indexed by the slot 0…T−1.
Forces next: we now hold input vectors of shape (B, T, d), but nothing tells us what the model should predict — there are no labeled targets. Lesson 03 must manufacture (input, target) pairs from raw text.

The plan

Four moves. (1) Pin down exactly why raw ids are a bad input — the fake ordering a net would exploit. (2) Introduce the embedding table as a differentiable lookup, equivalent to a one-hot matrix multiply but far cheaper, and see how backprop pulls similar tokens together. (3) Expose the order-blindness of what comes next — attention is a set operation — and fix it with a positional embedding added on top. (4) Sum the two tables into the input embedding, the (B, T, d) tensor the blocks eat. Then drive the lookup yourself and watch position break the symmetry.

1 · The problem with raw ids

Suppose our tiny tokenizer assigned ids like this: cat = 2, dog = 3, the = 5, runs = 1. Handing the sequence [5, 2, 1] ("the cat runs") straight to a network invites two silent errors.

Magnitude becomes meaning. The first layer computes something like w · id + b. To the model, "the" (5) is five times "runs" (1); a token that happened to be assigned id 50,000 would dominate the arithmetic entirely. But id size is an accident of the tokenizer's merge order — it encodes nothing. The network has no way to know that and will happily fit spurious patterns to it.

A fake ordering, a fake geometry. Integers live on a line: 2 is between 1 and 3, and 3 is closer to 2 than to 5000. That one-dimensional order is imposed on the vocabulary for free, and it is wrong. "cat" (2) and "dog" (3) being numerically adjacent is luck; "cat" (2) and "the" (5) being far apart is also luck. Any notion of "these two tokens are related" that the model could exploit is exactly the notion the raw ids destroy.

What we want is the opposite: a representation in a rich, many-dimensional space where distance encodes similarity, where every token starts symmetric (no accidental magnitude), and — crucially — where the representation can be shaped by training, because we do not know in advance which tokens should be near which. That is precisely an embedding.

Why not one-hot instead?

One honest fix is a one-hot vector: token i becomes a length-V vector that is 1 in slot i and 0 elsewhere. That kills the fake ordering and magnitude — every token is equidistant. But it is sparse and gigantic (length 50,257), and it is fixed: every pair of tokens is exactly as dissimilar as every other, so it can never express that "cat" is closer to "dog" than to "the." We need something dense, small, and learnable. The embedding is one-hot's smarter sibling — and, as we will see, it is literally a one-hot lookup in disguise.

2 · The embedding table: a learnable lookup

An embedding is a single matrix E of shape (V, d) — one row per vocabulary token, each row a vector of d real numbers. Turning an id into a vector is nothing more than reading that row:

embed(i) = E[i] E has shape (V, d), row i is token i's vector

In a small demo we might take V = 6 and d = 3; in GPT-2 the table is (50{,}257, 768), and in GPT-3 it is (V, 12{,}288). The table is initialized to small random numbers and — this is the whole point — its entries are parameters. They are optimized by gradient descent alongside every other weight in the model. The id is no longer a number the arithmetic touches; it is only an index that selects which trainable vector to use.

The lookup is a matrix multiply in disguise

Why is a "lookup" a legitimate differentiable operation that backprop can flow through? Because selecting row i is identical to multiplying a one-hot row vector by the table:

onehot(i) · E = E[i] where onehot(i) is (1, V) and E is (V, d) → (1, d)

A one-hot vector times a matrix picks out exactly one row. So the embedding layer is a fully connected linear layer whose input is a one-hot code — mathematically the same, just implemented as an index instead of a wasteful multiply-by-mostly-zeros. That equivalence is what makes it trainable: gradients flow back to the selected row and nudge it, leaving the untouched rows alone on that step.

Shapes: from ids to vectors

In practice we process a batch of B sequences, each of length T tokens. The ids arrive as an integer tensor of shape (B, T); the lookup replaces each id with its d-vector, appending a dimension:

ids            # (B, T)      integers in [0, V)
tok_emb = E[ids]              # (B, T, d)  each id -> its row
# equivalently, with a batch of 8 sequences of 4 tokens and d = 256:
# (8, 4)  ->  (8, 4, 256)

Nothing about the shape is mysterious: one number in becomes one d-vector out, per position. The book's demo uses d = 256 with the real BPE V = 50{,}257, so a batch of 8 sequences × 4 tokens becomes an (8, 4, 256) tensor — every token now a 256-dim vector.

How training makes similar tokens cluster

At initialization the rows are random noise — "cat" and "dog" are as far apart as any other pair. But training relentlessly rewards useful geometry. Whenever "cat" and "dog" can be swapped in a sentence and the next-token prediction stays sensible, the gradient that improves the loss for one nudges its row in a direction that also helps the other; over billions of tokens their vectors drift toward each other. Words that appear in similar contexts end up with similar vectors — the distributional hypothesis, made mechanical.

This is the same phenomenon Word2Vec made famous: in a well-trained embedding space, directions carry meaning, so the arithmetic king − man + woman ≈ queen roughly holds. We do not hand-craft any of this. We only provide the table and let the loss carve it. The takeaway for the chain: the embedding is what gives "meaning" a surface a gradient can shape. Raw ids had no such surface.

3 · Order-blindness: why we need position

There is a second problem hiding one lesson ahead. The engine that will consume these vectors — self-attention (Part II) — treats its input as a set, not a sequence. Each output position is a weighted blend of all the token vectors, and those weights come only from how the vectors match each other, never from where they sit. Permute the input tokens and you permute the outputs identically; the computation itself has no idea which token came first.

Concretely, to raw attention the sentences "dog bites man" and "man bites dog" are built from the same three vectors, so they produce the same bag of context vectors — even though their meanings are opposite. An order-blind model cannot tell a subject from an object, cannot know that "not good" differs from "good not." Language is not a set; word order is information, and right now we are throwing it away.

The fix is direct: give the model a second signal that depends only on the slot a token occupies. Build a positional embedding table P of shape (context_length, d) — one learnable vector per position 0, 1, 2, …, context_length−1 — and look it up by position index just as we looked up tokens by id:

pos_emb = P[0..T−1] P has shape (context_length, d) → the used slice is (T, d)

Here context_length is the longest sequence the model supports (1,024 for GPT-2); if the input is longer we must truncate. The positional vectors have the same width d as the token vectors, on purpose — because we are about to add them.

Absolute vs. relative positions

scheme	idea	who uses it
absolute, learned	a distinct trainable vector for each slot 0, 1, 2, …; the model discovers what "position 7" should mean.	GPT-2, and what we build here.
absolute, fixed	hand-designed sinusoids of varying frequency — no parameters, extends to unseen lengths.	the original Transformer.
relative (e.g. RoPE)	encode "how far apart" two tokens are rather than their absolute slots; generalizes better to longer sequences.	modern LLMs → see cross-ref.

We use GPT-2's absolute, learned scheme: a plain (context_length, d) lookup, optimized during training exactly like the token table. Relative schemes are the modern default and change what "position" even means at scale; that is a story for cs336 · 03 The modern recipe, which derives RoPE. For building a faithful GPT-2, absolute-learned is exactly right.

4 · The input embedding = token + position

Combine the two tables by adding them, element-wise, position by position. Because both have width d, the sum is again width d — the shape sails straight through:

input_emb = tok_emb + pos_emb (B, T, d) + (T, d) → (B, T, d)

The (T, d) positional slice is broadcast across the batch — the same position-7 vector is added to the 7th token of every sequence — so each token's final vector now encodes both what it is (the token row) and where it is (the position row). In pseudo-code the entire front end of a GPT is four lines:

tok_emb   = E[ids]                      # (B, T, d)   what each token is
positions = arange(T)                   # (T,)        [0, 1, 2, ..., T-1]
pos_emb   = P[positions]                # (T, d)      where each token sits
input_emb = tok_emb + pos_emb           # (B, T, d)   what + where

That (B, T, d) tensor — token identity plus position, fused into one vector per slot — is the input every transformer block consumes for the rest of the network. From here on, "the input" means this sum, never the raw ids.

Why add, not concatenate?

Adding keeps the width at d instead of doubling it to 2d, so no extra parameters ride downstream, and it lets the model superpose "what" and "where" in the same coordinates — a token can learn to write its identity into some directions and read position off others. It works because both signals are learned into a shared d-dimensional space, so their sum is a coherent point the next layer can disentangle.

5 · Watch the lookup, and watch position break the symmetry

The widget below holds a fixed 6-token vocabulary, each token a seeded d = 4 vector rendered as a strip of colored cells (a heatmap of its values). Pick a short id sequence with the buttons; the widget performs the lookup, pulling each token's row into a (T, d) grid — that is tok\_emb. Toggle + positional to add the per-slot vector P[t] on top, producing the input embedding. Watch the KPI "same token → same vector?": with position off, the same id in two slots gives byte-for-byte identical strips (a bag of words); with position on, the identical token now differs by slot — order has entered the representation. The shuffle order button reorders the sequence: with position off the multiset of vectors is unchanged (the set-symmetry attention suffers from), and with position on every slot's vector changes.

Lookup + position

Proves two things at once: the embedding is a learnable lookup that turns each id into a shapeable d-vector (top table), and the positional add is what breaks the set-symmetry — with it OFF the same token is the same vector everywhere (bag of words); with it ON, identical tokens at different slots become different inputs.

token table (V, d)

(6, 4)

input tensor (T, d)

—

same token → same vector?

—

position signal

Failure modes & checklist

Failure modes

Feeding raw ids to a dense layer. The net reads id-magnitude and the accidental vocabulary ordering as structure. Signal: nonsensical sensitivity to how tokens were numbered; performance changes if you renumber the vocab.
Omitting positional embeddings. The model becomes order-blind — "man bites dog" ≡ "dog bites man." Signal: it cannot use word order; scrambled inputs score the same.
Mismatched widths. Token and positional tables must share d; otherwise the add is undefined.
Sequence longer than context_length. No positional row exists for slot ≥ context_length; you must truncate (or the lookup errors).
Concatenating instead of adding without adjusting downstream. Doubles the width to 2d and breaks every shape that follows.

Checklist

Token table E is (V, d), learnable; the id only indexes a row.
Lookup E[ids] maps (B, T) → (B, T, d).
Positional table P is (context\_length, d), indexed by slot 0…T−1.
Input = tok\_emb + pos\_emb, same width d, shape (B, T, d).
Both tables are optimized by backprop (GPT-2: absolute, learned).

Checkpoint

Try it

Build a sequence that repeats one token in two slots — for example press the, then a different word, then the again. With + positional OFF, note that the two "the" strips are pixel-for-pixel identical and the KPI reads yes — that is the bag-of-words world attention lives in. Now turn positional ON: the two "the" strips diverge and the KPI flips to no. Finally press shuffle order with position OFF and then ON: off, the collection of vectors is merely reordered (same multiset); on, each slot's vector genuinely changes. Which of the three properties from lesson 00 — accurate, trainable, steerable — does the learnable table protect, and which does position protect?

Where this points next

We have finally built the model's true input: an (B, T, d) tensor where each token is a dense, trainable vector that also knows its place in the line. The architecture from here on is just transformations of this tensor. But notice what we still lack — a reason to change any of these numbers. The embedding table is initialized to noise and will stay noise unless something tells the model it is wrong. Training needs (input, target) pairs, and we have inputs with no targets. The next lesson finds targets for free, hiding in the text itself: the token that comes next is the label. That is lesson 03, The training signal — a sliding window.

Takeaway

Raw ids are arbitrary labels — their magnitude and ordering are accidents a network would misread. The fix is an embedding table E of shape (V, d): a differentiable lookup (mathematically a one-hot × matrix, cheaper as an index) whose rows are parameters, so backprop pulls semantically similar tokens together — king − man + woman ≈ queen. Because the attention that follows is a set operation, we add a second learnable positional table P of shape (context\_length, d), indexed by slot, so the same token at two positions becomes two different vectors. Their sum, input\_emb = E[ids] + P[0..T−1] of shape (B, T, d), is the tensor every transformer block consumes. Embeddings give meaning a gradient can shape; position restores the order that a set discards.

Interview prompts

Why can't you feed integer token ids directly into a neural network? (§1 — the first layer multiplies by them, so it reads id-magnitude and the accidental vocabulary ordering as meaning, even though both are artifacts of the tokenizer.)
In what sense is an embedding layer "just a lookup," and why is that differentiable? (§2 — selecting row i equals onehot(i)·E, a matrix multiply; gradients flow back to the selected row, so the table is trainable — a lookup implemented as an index rather than a multiply by zeros.)
What are the shapes going into and out of the token embedding? (§2 — ids (B, T) integers → (B, T, d) vectors; each id becomes its d-dimensional row.)
Why do semantically similar tokens end up with similar vectors? (§2 — training rewards useful geometry; tokens appearing in similar contexts get nudged in similar directions over billions of examples, so distance ends up encoding similarity — the distributional hypothesis, e.g. king − man + woman ≈ queen.)
Why does self-attention need positional information at all? (§3 — attention is a set operation: it blends token vectors by how they match, not by where they sit, so without a position signal "dog bites man" and "man bites dog" are identical inputs.)
How is the positional signal combined with the token embedding, and why that way? (§4 — a (context\_length, d) table is looked up by slot and added to the token vector; adding keeps width d (no extra downstream params) and lets "what" and "where" superpose in one shared space.)
What is the difference between absolute-learned and relative positional embeddings? (§3 — absolute-learned (GPT-2) gives each slot its own trainable vector; relative schemes like RoPE encode distance between tokens and generalize better to unseen lengths — see cs336/03.)

Companion reads: cs336 · 03 The modern recipe derives RoPE and the relative-position schemes modern LLMs prefer; cs336 · 02 The Transformer counts the embedding parameters (V·d) inside the full parameter budget.