LLMs & generative AI in recommendation
Every team is being asked "where does the LLM go in our recommender?" The honest answer is mostly "not in the hot path." A single LLM call costs 10²–10⁴× a DLRM forward pass, so the interesting question is not whether LLMs help but at which layer the economics let them help at all. This lesson is a hype filter: four concrete roles, the cost arithmetic that pins each one to a stage of the funnel, and the failure modes that make the naïve version worse than the system it replaced.
First principles: the cost wall decides the layer
Recommenders are a cascade — retrieval narrows millions to thousands, ranking thousands to hundreds, re-ranking hundreds to the handful you show (lessons 02 and 03). Each stage operates at a different QPS-per-candidate, and that single number — how many model invocations per request — sets the per-call budget. A retrieval stage scoring 10⁶ candidates at 10⁵ QPS has a budget of microseconds and fractions of a cent per request. An LLM call is tens of milliseconds and a fraction of a cent per call. Multiply by candidates and the retrieval bill is absurd; at the re-rank stage, where you score 20–200 items once per request, it is merely expensive.
So the layer where an LLM lives is not a modeling choice — it falls out of arithmetic. Let a DLRM forward pass cost c_d ≈ 10⁻⁶ s and an LLM call c_l ≈ 5·10⁻² s. Then c_l / c_d ≈ 5·10⁴. Putting an LLM at retrieval (10⁶ candidates) versus re-rank (10² candidates) is a four-order-of-magnitude difference in spend for the same QPS:
The whole lesson is an elaboration of that diagram. Four roles, ordered from "most real today" to "most overhyped": evaluation, generative retrieval, re-ranking/reasoning, and AIGC content. For each, the test is the same — does the LLM do something a cheaper model provably cannot, and does the layer's budget allow it?
| LLM role | Where in funnel | Added latency / cost | What it uniquely buys | Chief risk |
|---|---|---|---|---|
| LLM-as-judge / user simulator | Offline eval (no serving path) | Seconds/call, batchable, $$ per eval run not per request | Graded labels where humans are scarce; long-horizon satisfaction proxy | Judge bias & miscalibration; circular if judge ≈ policy |
| Generative retrieval (semantic IDs) | Replaces retrieval index | Autoregressive decode per request; tight latency budget | No separate ANN index; cold items via semantic codes | Hallucinated/invalid IDs; decode latency; hard to update freshly |
| LLM re-ranker / reasoner | Re-rank (shortlist of ~10²) | ~1 LLM call/request, 30–100 ms | Reasoning over a small set; explanations; conversational/agentic | Cost ceiling caps list size; position/verbosity bias in ordering |
| AIGC content & synthetic data | Offline / async pipeline | Amortized; not in request path | Creative variants, cold-start content, query augmentation | Hallucinated items; feedback-loop homogenization; provenance |
Role 1 · LLM-as-judge and generative user simulators
This is the most production-real role today, precisely because it lives off the serving path: latency and per-request cost don't matter when the LLM runs in a nightly batch. The motivating problem is the one lesson 08 opened with — every offline metric is a summary statistic on logged predictions, biased toward items the old policy already surfaced, and human relevance labels are expensive and slow. An LLM judge promises graded relevance labels at near-zero marginal cost: feed it (query, item) or (user history, recommendation) and ask for a 1–5 relevance score or a pairwise preference.
Two distinct uses, often conflated:
- LLM-as-judge for relevance/quality. Replace or augment human raters that produce the graded labels feeding NDCG@K. Useful for screening many ranker variants before they earn A/B traffic — exactly the "filter, not decision" role lesson 08 assigns to offline metrics.
- Generative user simulators for off-policy evaluation. Instead of importance-weighting logged rewards (the IPS / doubly-robust machinery of lesson 08), simulate a user: prompt an LLM with a persona and history, show it the new policy's recommendations, and read off simulated clicks/dwell. This can score items the old policy never surfaced — the central blind spot of logged-data offline metrics — but only as well as the simulator models real users, which is the whole catch.
Be skeptical and concrete. An LLM judge is itself a model with systematic biases, and treating its scores as ground truth is how teams ship regressions confidently:
Where it complements classical evaluation, and where it does not: an LLM judge is a cheap, scalable proxy for human relevance labels, so it slots in exactly where NDCG-style graded labels are scarce and helps you prune the candidate set of A/B tests. It does not replace the A/B test — it has no access to the counterfactual of real user behavior, no measurement of long-term retention, and inherits none of the randomization that makes A/B tests immune to position bias. The senior framing matches lesson 08's spine: LLM judge filters, A/B test decides. A judge that disagrees with your A/B outcome is telling you the judge is miscalibrated for your surface, not that the A/B was wrong.
Role 2 · Generative retrieval and semantic IDs
The two-tower + ANN paradigm of lessons 02/03 retrieves by matching: embed the query, embed every item, find nearest neighbors in a maintained index. Generative retrieval flips this — instead of looking items up, the model generates the identifier of the item to retrieve, autoregressively, like a language model emitting tokens. This is the TIGER line of work, and the key enabling trick is the semantic ID.
A raw item ID (e.g. item_8675309) is an atomic, meaningless token — two similar items have unrelated IDs, so a model must memorize each one. A semantic ID instead factorizes an item's content embedding into a short sequence of discrete codes via residual-quantized VAE (RQ-VAE): quantize the embedding to the nearest codeword in a learned codebook, take the residual, quantize again, and repeat for L levels. An item becomes a tuple of codewords:
item → (c₁, c₂, …, c_L), c_ℓ = argmin over k of ‖ r_(ℓ-1) − e_k^(ℓ) ‖², r_ℓ = r_(ℓ-1) − e_(c_ℓ)^(ℓ)
The codes are coarse-to-fine semantic: items sharing c₁ are in the same broad cluster, sharing (c₁,c₂) a finer one. Now a sequence model trained on user histories (a transformer over the user's past semantic-ID sequence) learns to generate the next item's semantic ID token by token, with retrieval being a beam search over the code vocabulary. The probability factorizes autoregressively:
P(item | history) = ∏ over ℓ=1..L of P(c_ℓ | c_(<ℓ), history)
| Two-tower + ANN (lessons 02/03) | Generative retrieval (TIGER-style) | |
|---|---|---|
| Retrieval mechanism | Embed query, nearest-neighbor search over item index | Beam-search decode item's semantic-ID sequence |
| Separate index? | Yes — ANN index (HNSW/IVF) built & maintained | No — "index" is the model's learned weights |
| Cold items | Need a content-derived embedding to be inserted | Get a semantic ID from content immediately; sequence model can emit it without retraining its own row |
| Adding/updating items | Cheap — insert a vector into the index, near-real-time | Hard — new code assignments / knowledge may need retraining; freshness is the weak point |
| Latency | Sub-ms ANN lookup, very mature | L autoregressive decode steps × beam width — heavier, the open problem |
| Invalid results | Impossible — every neighbor is a real item | Possible — beam can emit a code tuple matching no item (hallucinated ID); needs constrained decoding / a trie of valid sequences |
| Maturity | Battle-tested at 10⁵-QPS scale | Promising research; few full-scale production deployments |
Honest verdict: generative retrieval is genuinely interesting because it collapses "embedding model + ANN index + nightly index rebuild" into one trained artifact and handles cold items naturally through shared semantic codes (the cold-start bridge of lesson 15, now baked into the ID). But the two killers are decode latency (autoregressive generation is inherently serial where ANN is a single parallel lookup) and invalid/hallucinated IDs (you must constrain decoding to a trie of real items, or you retrieve nothing). Freshness is the third: an ANN index ingests a new item in seconds, while a generative model may need retraining to reliably emit a newly-popular item's code. For most production systems in 2026 this is a research bet, not a replacement for the two-tower stack — promising, not proven.
Role 3 · LLM re-rankers and reasoning
This is where the cost arithmetic finally permits a real LLM in the request path — barely. At the re-rank stage you have a shortlist of ~10²; one LLM call per request is ~50 ms and a fraction of a cent, which a re-rank budget can absorb where a retrieval budget cannot. Three flavors:
- Listwise LLM re-rank. Hand the LLM the user context and a shortlist of candidate titles/features and ask it to reorder them, reasoning over the set (complementarity, diversity, intent) in a way a pointwise scorer cannot. This is the natural home: small list, one call, reasoning genuinely adds something.
- Explanations. Generate the "because you watched X" rationale (lesson 21). Critically, the explanation should be generated from the real reason the item ranked (its features/attribution), not invented post-hoc — an LLM left to rationalize will fabricate a plausible-sounding reason that does not match why the model actually picked the item.
- Conversational / agentic recommendation. The user states intent in natural language ("something cozy but not a rom-com"); the LLM interprets, optionally calls retrieval as a tool, and recommends conversationally. The latency wall keeps the LLM as the orchestrator over fast retrieval tools, not as the retriever itself.
The senior point: an LLM at re-rank is justified only when reasoning over the set, multimodal understanding (lesson 14), or natural-language interaction buys something a distilled DLRM cannot. Often the right move is the reverse — distill the LLM's judgments into a cheap student model that runs at rank-stage cost, getting most of the quality at none of the latency. Use the LLM to generate training signal, not to serve it.
Role 4 · AIGC content and synthetic data
Generative models can produce content, not just rankings: ad creatives and thumbnails (diffusion), synthetic descriptions for cold items, query rewrites and feature augmentation. This lives in offline/async pipelines, so cost is amortized — the appeal is real. The risks are subtler than latency and they compound over time.
- Generative creatives / ads. Auto-generate creative variants, then let the bandit/exploration machinery of lesson 15 pick winners. Genuinely useful: cheap variant generation feeds exploration.
- Synthetic cold-start content. A cold item with sparse metadata gets an LLM-generated description/tags, giving the content tower (lesson 15) something to embed before any interactions exist.
- Query / feature augmentation. Expand sparse queries, generate synonyms, fill missing attributes — classic data augmentation, now generative.
Interactive · where does the LLM belong?
Pick a funnel stage to drop an LLM into, set your traffic and the per-call cost/latency, and the calculator computes the LLM invocations per request, the resulting $/day, the added p99 latency, and a verdict. Retrieval and ranking score one LLM call per candidate (10⁶ and 10³ respectively) — the economics scream. Re-rank is the trick: a single listwise call over the whole shortlist, so it is one call per request regardless of list size. Eval is offline. Try putting it at retrieval and watch the bill go to the billions.
Crosscutting: train–serve, freshness, and the distillation escape hatch
Two systems concerns recur across all four roles. Freshness: LLMs have a knowledge cutoff and are expensive to retrain, so anything time-sensitive (a new item, a breaking trend, a price change) is a weak spot — which is exactly why generative retrieval struggles with new items and why AIGC content can go stale. The fix is to feed fresh facts at inference (retrieval-augmented prompting) rather than baking them into weights. Train–serve skew: an LLM used offline to generate labels or content but absent at serving creates a gap — the production ranker must be trained to consume what the LLM produced, and you must monitor that the LLM's offline judgments still predict online outcomes. The cleanest reconciliation is the distillation escape hatch: use the expensive LLM offline to produce a teacher signal (judgments, rankings, synthetic labels), then train a cheap student that serves at DLRM cost. You get the LLM's reasoning at the latency of a forward pass, and you never put the LLM in the 10⁵-QPS hot path at all.
Interview prompts you should be ready for
- "Where in the recommender stack would you actually put an LLM, and why not retrieval?" (Lead with the cost arithmetic: an LLM call is ~10²–10⁴× a DLRM forward pass, and per-request LLM cost scales with candidates scored. Retrieval scores 10⁶/request → impossible; re-rank scores 10²/request → one call, affordable; eval/content are offline → cost amortized. The layer falls out of arithmetic, not cleverness.)
- "You want to use an LLM-as-judge to label relevance. What could go wrong, and how do you trust it?" (Position bias, verbosity/length bias, self-preference — fatal and circular if judge = generator — and poor calibration of the numeric score. Non-negotiable mitigation: validate against a human-labeled gold set and report agreement (κ / correlation) before gating launches; average over presentation orders; check self-consistency across samples. The judge filters; the A/B test still decides.)
- "Contrast generative retrieval with two-tower + ANN." (Two-tower matches via nearest-neighbor over a maintained index; generative retrieval decodes an item's semantic ID autoregressively via RQ-VAE codes, so the 'index' is the model's weights. Pros: no separate index, natural cold-item handling through shared codes. Cons: serial decode latency vs parallel ANN lookup, hallucinated/invalid IDs needing constrained decoding, and poor freshness — adding an item is a near-real-time index insert vs a possible retrain.)
- "What is a semantic ID and why does RQ-VAE help?" (Factorize an item's content embedding into a short coarse-to-fine sequence of discrete codes via residual quantization. Items sharing prefixes are semantically related, so a sequence model can generalize across items and emit a cold item's ID from content alone — instead of memorizing an atomic, meaningless raw ID.)
- "Your team wants an LLM to re-rank the top 100. What are the risks and the cheaper alternative?" (Risks: position/verbosity bias leaking into the ordering, p99 tail blowing the latency budget, hallucinated/dropped items — constrain output to a permutation of input IDs, randomize order, keep a classical fallback with a circuit breaker. Cheaper alternative: distill the LLM's listwise judgments into a student model that serves at rank-stage cost.)
- "What is the danger of training on AIGC content, and how do you mitigate it?" (The autophagy / feedback loop: model trains on its own generations → distribution narrows → population-level mode collapse, catalog homogenization, amplified filter bubble. Mitigate with provenance tagging of synthetic content, a cap on the synthetic fraction of training data, a floor of real human signal, and validating generated items against the real catalog to kill hallucinations.)
- "When does an LLM genuinely beat a cheaper model, versus when is it hype?" (Real: reasoning over a small set at re-rank, natural-language/conversational intent, multimodal understanding, generating labels/content offline where humans are scarce. Hype: anything in the high-QPS retrieval/rank hot path where a distilled or two-tower model gets ~the same quality at 10³–10⁴× lower cost. The test is whether the LLM does something a cheaper model provably cannot AND the layer's budget allows it.)