CLIP & the text-conditioning interface
We have a latent space (lesson 19) and a way to steer the score with a condition (lesson 18). The missing piece: turn a sentence into that condition, and inject it into the denoiser. That is CLIP plus cross-attention.
The job
Guidance (lesson 18) needs a condition vector y; everything downstream is “add to the score.” For text-to-image, y must be a numeric encoding of a prompt that lives in a space the image model understands. Two questions: (1) how do you train a text encoder whose outputs are already aligned with image content? (2) how does that encoding physically enter the UNet/DiT? CLIP answers the first; cross-attention answers the second.
CLIP: align text and images by contrast
CLIP (Radford et al. 2021) is two encoders trained jointly:
- Image encoder — a ViT (or ResNet) mapping an image to a vector.
- Text encoder — a Transformer (GPT-2-style) mapping a caption to a vector.
Both project into a shared embedding space and are L2-normalized. The training signal is purely contrastive: in a batch of N (image, caption) pairs, the N true pairs should have high cosine similarity and the N²−N mismatched pairs low. There are no class labels — the supervision is “which caption goes with which image,” harvested free from 400M web pairs.
The symmetric InfoNCE loss, derived from the matrix
Compute all pairwise similarities into an N×N logit matrix, scaled by a learned logit scale 1/τ (CLIP actually learns a log-scale parameter s and uses 1/τ = \exp(s), clamped for stability, so the temperature is τ = \exp(−s)):
The correct match for row i is column i — the diagonal. So treat each row as an N-way classification (image i picking its caption) and each column likewise (caption j picking its image), and average the two cross-entropies:
This is the “symmetric cross-entropy” / InfoNCE the canonical CLIP pseudocode computes (loss_i over axis=0, loss_t over axis=1, averaged). Read the loss as a question asked of every row and every column: out of the N captions on offer, which one is the true partner of this image? The softmax turns the row of similarity scores into a probability distribution over candidates, and cross-entropy says “put the mass on the diagonal entry.” The only way to win the row game and the column game at once is for matched pairs to be the most similar things in the batch — which is exactly the shared space we wanted. The temperature τ controls how peaked that softmax is — small τ (large 1/τ) makes the model react sharply to a near-miss negative; CLIP learns it rather than fixing it.
Claim: a loss that only ever asks “which caption matches this image?” is enough to force text and images into one shared semantic space.
- One score per pair. Cosine similarity is just “how aligned are these two vectors,” computed for all N×N image-caption combinations. The diagonal is the true pairs; everything off-diagonal is a mismatch.
- Turn scores into a bet. Softmax along a row converts that image’s N similarity scores into a probability over the N captions — the model is being asked to bet which caption belongs to it.
- Penalize a bad bet. Cross-entropy against the diagonal pays out only when the bet lands on the true caption. To lower the loss the model has just one lever: make the matched pair more similar than every distractor in the batch.
- Both directions at once. Doing the same down each column forces captions to also pick the right image. Satisfying both pulls a pair’s two vectors together and shoves unrelated pairs apart, all in the same geometry.
Central point. Nobody ever tells the model what an image “means” — just which caption it goes with — yet winning that matching game over 400M pairs leaves text and images sharing one space where “closeness” is “same meaning.”
How the text vector enters the denoiser: cross-attention
The denoiser is a UNet (SD) or DiT (lesson 8) over image latents. Text conditioning is injected via cross-attention layers interleaved with the existing self-attention. The crisp self-vs-cross distinction interviewers want, made precise:
| Self-attention | Cross-attention | |
|---|---|---|
| Q (query) | image tokens | image tokens |
| K, V (key/value) | image tokens (same sequence) | text tokens (other sequence) |
| Learns | relationships within the image (global coherence) | alignment between image regions and words |
Strip away the matrices and the move is simple: each image patch asks a question (its query), compares it against every text token’s label (the keys), and the better a word matches the question the more of that word’s content (its value) the patch absorbs. The softmax(QKᵀ/√d) is just the “how strongly do I listen to each word” weights, normalized to sum to one; multiplying by V blends the chosen words’ content and adds it back to the patch. So the attention map is literally “which word does this patch listen to” — the mechanism by which “a red hat” makes the hat-region patches pull from the “red” token. SD uses CLIP’s per-token hidden states (the sequence, not just the pooled vector) as K/V, so spatial words can land on spatial regions.
Claim: cross-attention is each image patch shopping the prompt for the words that concern it, and copying in a blend of their meaning.
- The patch asks. Q = WQ·(image) turns each image patch into a query — “what kind of content am I looking for?”
- The words answer. K = WK·(text) and V = WV·(text) give every prompt token a searchable label (key) and a payload of meaning (value).
- Score the match. Q Kᵀ scores how well each patch’s question fits each word’s label; the softmax turns those scores into listening weights that sum to one (a patch divides its attention across words).
- Copy in the meaning. Multiplying the weights by V mixes the relevant words’ content and adds it to the patch — the hat patch ends up carrying “red.”
Central point. Conditioning isn’t a global knob; it’s a per-patch lookup into the prompt, so different regions of the image can be steered by different words at once.
CLIP-feature pros and cons (a stock interview question, with reasons)
| Strength | Why |
|---|---|
| Strong generalization, zero-shot | trained on 400M diverse web pairs; the shared space already “knows” photorealistic / watercolor / cyberpunk as directions |
| One space for both modalities, no extra training | text and image are directly comparable by cosine — drop-in conditioning, also enables CLIP-score eval (lesson 21) and IP-Adapter (lesson 23) |
| Weakness | Why |
|---|---|
| Weak on compositional / relational / counting prompts | contrastive matching rewards bag-of-concepts alignment, not syntax — “A but not B”, “three cats”, “left of” degrade |
| Captions were short and simple | training text was mostly “a photo of a ___”; long art-direction prompts hit a bottleneck |
This is precisely why production models bring in a T5 text encoder — a pure language model with real syntactic structure, so it carries the negation and composition CLIP confuses. Imagen went further and used T5-XXL alone (no CLIP); SD3 (lesson 15, lesson 22) keeps CLIP and adds T5, concatenating their outputs as the K/V for cross-attention so the denoiser gets both “visual-concept alignment” (CLIP) and “sentence structure” (T5). Either way the lesson is the same: contrastive vibe-matching needs a syntactic partner.
Interactive · the contrastive matrix & temperature
Five toy (image, caption) pairs sit in a shared 2D space. The heatmap is the similarity matrix; CLIP’s loss wants the diagonal hot and everything else cold. Drag the embeddings to create a “hard negative” (two pairs almost overlapping) and watch the diagonal contrast collapse; drop the temperature τ and watch the softmax sharpen the model’s confidence on the diagonal.