The funnel — retrieve, rank, re-rank
Why production ranking is never one model. The latency, recall, and precision budget that forces the cascade. The first thing you draw in any system-design interview.
From first principles: why not a single model?
Imagine you have a perfect oracle: a single function f(user, item, context) → score that returns exactly the relevance you want. You run it on every (user, item) pair and pick the top K. Done.
The reason that fails in production is arithmetic, not modelling. Suppose:
- Catalogue size N = 10⁹ items (YouTube videos, products, ads).
- p99 serving budget: 100 ms end-to-end. The ML budget is maybe 50 ms of that — the rest is network, feature fetch, business logic, rendering.
- The "good" ranking model is a deep network with hundreds of features and embedding lookups. A reasonable per-item scoring cost: ~100 µs. (This is generous — production DLRM-class models with sparse features and cross-feature lookups can be 100–500 µs.)
Score every item: 10⁹ × 100 µs = 10⁵ seconds ≈ 28 hours. Per request. The math doesn't allow one model.
So you cascade. A cheap model — typically dot product between two precomputed embeddings — over the full catalogue produces a few hundred or a few thousand candidates. The expensive model scores only those. Optionally a third stage applies business logic (diversity, freshness, advertiser/policy filters) on the top dozens.
What each stage optimizes for — and why they can't be the same metric
This is the trap interviewers love to set: "if every stage optimizes for the same thing, why have stages?" The answer is that they don't.
| Stage | Metric | Why this one |
|---|---|---|
| Candidate gen | Recall@K₁ | You only have to not drop good items. Ranking is the ranker's job. False negatives here are fatal; false positives are cheap (they get filtered later). |
| Ranking | NDCG / AUC / calibrated CTR | You have to order the K₁ candidates well, and (for ads) emit a probability you can multiply with a bid. Precision matters because only the top dozens are shown. |
| Re-ranking | Long-term engagement, diversity, business constraints | The ranker scores items in isolation; users see a list. Re-ranking accounts for inter-item effects, freshness, exploration, ad load, policy violations. |
A way to internalize this: the candidate generator is allowed to be wrong about ordering as long as good items are in the K₁. The ranker is allowed to ignore items that didn't make it into K₁. The re-ranker is allowed to ignore most ranking detail and worry about the top-of-list user experience. Each stage narrows, and each stage gets richer features and a more expensive model as it does.
Four stages, not three — where coarse-rank (粗排) comes from
The three-box drawing above is the right mental model, but it hides a seam. Look at the count gap between the two scored stages: retrieval hands the ranker K₁ ≈ 10³ candidates, and the heavy ranker can only afford to score ~10² in budget. Something has to cut 10³ → 10² before the expensive model runs. In every industrial system that "something" is a fourth, distinct stage that the textbook three-box picture quietly skips: coarse-rank (粗排), sitting between retrieval (召回) and fine-rank (精排).
So the canonical industrial cascade has four stages, each shedding roughly an order of magnitude:
The two end stages are unchanged from the three-box picture — retrieval and re-rank are the same. What got split is the middle: the old "ranking" box was secretly doing two jobs at two different price points, and production pulls them apart into 粗排 and 精排.
Why coarse-rank must exist as its own stage — the cost × count inequality
This is the part interviewers probe, and it is pure arithmetic. Give each stage a budget of ~10 ms of ML time and work out how many items each model can afford to score:
items a stage can score ≈ (stage latency budget) ÷ (per-item model cost)
| Stage | Model | cost / item | Items affordable in ~10 ms | Input it must handle |
|---|---|---|---|---|
| Retrieval (召回) | ANN over precomputed embeddings | ~1 µs sub-linear | 10 ms ÷ 1 µs = 10⁴ exact scans — and the ANN index makes even 10⁸ reachable sub-linearly | N = 10⁸ ✓ |
| Coarse-rank (粗排) | light twin-tower DNN (dot of two small towers) | ~10 µs | 10 ms ÷ 10 µs = 10³ | K₁ = 10³ ✓ (exactly enough) |
| Fine-rank (精排) | heavy DLRM / MMoE, hundreds of cross features | ~100 µs | 10 ms ÷ 100 µs = 10² | K₂ = 10² ✓ — but it would choke on K₁ = 10³ (needs 100 ms) |
Now make the gap concrete. Suppose retrieval returns K₁ = 10³ candidates and you tried to feed them straight into the fine ranker, skipping coarse-rank:
10³ items × 100 µs/item = 10⁵ µs = 100 ms — already over the entire end-to-end budget.
That is the whole argument. The fine ranker is 10× too expensive to run on the retrieval output. But you cannot fix it by retrieving fewer items: dropping K₁ to 10² would let the fine ranker keep up, yet it would gut recall — the retriever's ordering is too crude to trust its top-100, which is exactly why you retrieved 10³ in the first place. So you are squeezed from both sides:
The pattern generalizes into the one sentence worth memorizing: each stage spends compute inversely to how many candidates it sees — a cheap model on many items, an expensive model on few, and each step trades roughly 10× more candidates for roughly 10× less per-item cost so the product (total work) stays inside the latency budget at every stage:
| Stage | count × cost | ≈ total work |
|---|---|---|
| Retrieval (召回) | 10⁸ × ~0 (sub-linear ANN) | ~few ms |
| Coarse-rank (粗排) | 10³ × 10 µs | 10 ms |
| Fine-rank (精排) | 10² × 100 µs | 10 ms |
| Re-rank (重排) | 10¹ × O(ms) | ~few ms |
Notice the middle two rows land at the same total work (~10 ms) despite a 10× difference in both count and cost — that is the balance the cascade is engineered to hit, and it is why the count drops by ~an order of magnitude at each step rather than all at once.
Cross-links for the other three stages: the heavy fine-rank model class (DLRM / MMoE / multi-task heads) is lesson 04 (ranking models); the re-rank stage's list-aware quality is lesson 45 (semantic reranking), and its diversity / fairness constraints are lesson 20 (fairness & diversity).
Interactive · the latency budget
Drag the sliders to set the catalogue size, end-to-end latency budget, and per-item model cost. The widget tells you whether you can serve from one stage, two, or three — and where the bottleneck is.
The recsys ↔ ads parallel
The same funnel works for both. Naming and metrics differ, but the structure is one-to-one.
| Recommender (e.g. YouTube Home) | Search ads (e.g. Google Sponsored Search) | |
|---|---|---|
| Retrieve from | All ~10⁹ candidate items | All eligible ads matching the query (typically 10³–10⁵ after match-type and policy filters) |
| Retrieval signal | User × item affinity (two-tower) | Query × ad relevance (keyword + semantic match) |
| Ranker predicts | P(watch ≥ 30s), P(like), engagement | P(click), P(convert) — multiplied by advertiser bid |
| Re-rank applies | Diversity, freshness, creator caps | Auction, reserve, ad-load, policy |
| Optimizes | Long-term engagement | Long-term auction efficiency (≈ user clicks × advertiser ROI × revenue) |
The auction layer is the only thing ads adds. Everything else — retrieval, ranking, calibration, debiasing — is the same machine. This is why senior ads MLEs and senior recsys MLEs are roughly interchangeable; only the last layer of vocabulary differs.
Multi-objective ranking — the ranker rarely predicts one thing
In an interview you'll be expected to know that "the ranking model" almost never has a single output. Modern production rankers are multi-task: they predict several engagement signals simultaneously, then combine them into one scalar score for sorting.
For YouTube-style recsys, a paper-canonical example (Zhao et al. 2019, "Recommending What Video to Watch Next") predicts roughly:
- P(click) — did the user click the thumbnail?
- P(watch ≥ 10s) — past the bounce threshold?
- E[watchtime | click] — regression head, conditional on a click.
- P(like), P(share), P(subscribe) — explicit engagement.
- P(skip), P(dislike) — negative signals.
The final score is a hand-tuned (or learned-to-rank) combination, e.g. α·P(watch) + β·E[wt] − γ·P(skip) + …. The weights are policy decisions, not model outputs — and interviewers love to ask "who picks the weights and how?" The honest answer is some combination of online A/B testing and Pareto-frontier evaluation; there is no off-the-shelf calibration for "long-term user happiness".
For ads, the multi-task ranker predicts P(click) and P(conversion | click), since the auction's expected revenue is bid × P(click) × P(conv | click) (or cpc_bid × P(click) for click-based campaigns). One model produces both — multi-task heads share embeddings and are much more sample-efficient than two separate models.
Where the funnel breaks — and what production fixes look like
Three failure modes that show up in interviews because they show up in production.
| Failure | Symptom | Production fix |
|---|---|---|
| Retrieval ↔ ranker mismatch | The ranker is great offline but the funnel underperforms online: the items the ranker thinks are best aren't in K₁. | Co-train. Ranker-distillation into the retriever (the retriever's loss includes the ranker's score on hard negatives). Or, periodically, hard-mine ranker top-K items the retriever missed and add them as positives. |
| Position bias compounds across stages | The training data for the ranker is clicks on items the old retriever surfaced at top positions. New retrievals never get a fair shot because their items appear lower (or never). | IPS / examination-model debiasing (lesson 7). Explicit exploration slot. Counterfactual evaluation before launch. |
| Re-rank starves the ranker | Heuristics at the re-rank stage (diversity, ad-load) routinely override the top-ranked items, so improving ranking quality has near-zero online impact. | Audit re-rank overrides as if they were a separate model with their own metric. Move policy-as-code to a learned re-ranker (e.g., generative re-rankers, listwise transformers). |
Interview prompts you should be ready for
- "Design YouTube Home feed end-to-end." (Draw the funnel within 30 seconds. Then go deep on whichever stage the interviewer steers you toward.)
- "Why can't you train one neural network on (user, item, context) over the full catalogue?" (Answer in arithmetic: per-item cost × catalogue size vs latency budget. Do the multiplication out loud.)
- "Your ranker improves NDCG offline by 5% but the A/B test is flat. What happened?" (Probes: retrieval mismatch, position bias, the ranker improvement was on items the retrieval never surfaces, novelty bias, or the metric used offline doesn't match the production metric.)
- "How do you decide K₁ — the candidate-set size?" (Trade-off between recall and ranker cost. Empirically: plot recall@K vs K, pick the elbow, then verify the ranker latency budget. There's no closed-form answer.)
- "Why is there a coarse-rank (粗排) stage between retrieval and the fine ranker — why not just retrieve fewer items?" (Answer in the cost × count inequality: 10³ candidates × 100 µs ≈ 100 ms blows the budget, so the fine ranker is ~10× too expensive for the retrieval output; but shrinking K₁ to 10² to compensate guts recall because the retriever's ordering is too crude to trust its top-100. The only escape is an intermediate-cost model — a light twin-tower 粗排 — wide enough to score all 10³, sharp enough to pick the 10² for fine-rank. Bonus signal: name the consistency/distillation requirement — the coarse ranker must agree with the fine ranker or it silently caps quality; see lesson 19.)
- "What's the difference between the metric a ranker optimizes and the metric a feed optimizes?" (Probes: ranker outputs per-item probabilities; feed quality is a property of the list — diversity, freshness, ad-load. Different objects.)
- "For ads, the auction is at the end. Why not earlier?" (You can only auction items you have a calibrated CTR for. So auction is downstream of ranking. Retrieving cheaply and ranking carefully is what feeds the auction useful inputs.)