Learning to rank for search

Retrieval (lessons 42/43) hands you a few hundred candidates. Learning to rank (LTR) scores them with rich query–document features and produces the relevance ordering the user actually sees. The art is the feature vector and the choice of objective.

Where this sits in the cascade (lesson 01): retrieval answers "which ~500 documents could plausibly match?" cheaply, over the whole corpus. Ranking — this lesson — answers "what is the exact order of those 500?" using features too expensive to compute corpus-wide. The next lesson (45) adds a small, even-more-expensive neural rerank stage on the top ~50. Each stage spends more compute on fewer documents. LTR is the middle stage, and historically the one that moved the metric most per dollar.

The feature vector — the heart of classical LTR

A classical LTR model is a function s = f(x) where x is a feature vector for a single (query, document) pair, and s is a scalar score. You compute x for every retrieved candidate, score each one, and sort descending. Everything interesting lives in x. A production web-search ranker has hundreds to low-thousands of features; they fall into four families.

The features must be cheap enough to compute for a few hundred candidates within the ranking latency budget, but they can be far richer than retrieval's — retrieval features must be precomputable into an index over a billion documents; ranking features are computed on-the-fly for ~500. That asymmetry is the whole reason for a separate ranking stage. Term-proximity over the matched spans, field-level BM25, and a cross-feature like "query-length × doc-length" are all fair game in the ranker and impossible in the inverted index.

Labels — what is "correct"?

To learn f you need a target relevance for each (query, document). Two sources, with opposite trade-offs.

Graded human judgments (qrels)

Human raters assign a graded relevance label on an ordinal scale, classically 0–4: 0 = irrelevant, 1 = related, 2 = relevant, 3 = highly relevant, 4 = perfect/navigational. The set of judged (query, doc, grade) triples is called qrels (query relevance judgments — the term comes from the TREC evaluation tradition). Grades are expensive (rater time, guidelines, double-judging for agreement) but unbiased by presentation: a rater judges the document on its merits, not on where it happened to appear.

Pooling. You cannot judge every document for every query — that is billions of pairs. Instead you pool: run several ranking systems (or several configs), take the union of each one's top-k (say top-20), and send only that pool to raters. Anything no system surfaced is assumed irrelevant. Pooling makes judgment tractable but introduces a bias of its own — a future system that surfaces a genuinely great document no pooled system found gets no credit for it. You re-pool periodically to patch this.

Implicit clicks

Click logs are nearly free and arrive in the billions: every search session is a labeled-ish example. But clicks are biased labels, and conflating "clicked" with "relevant" is the classic junior mistake. The dominant distortion is position bias: a document at rank 1 gets clicked far more than the same document at rank 5, simply because users look at the top first. So a click confounds "this document is relevant" with "this document was shown high." There is also presentation bias (rich snippets, thumbnails draw the eye) and trust bias (users trust top results). All of this is lesson 07; the consequence for LTR is that you cannot feed raw clicks to a ranker without correction, or you train it to reproduce the previous ranker's ordering — a feedback loop, not learning.

The mature answer uses both: human judgments as the unbiased evaluation gold and for high-value head queries, and position-debiased clicks (IPS weighting, below) for the volume and tail coverage that judgments can never reach.

Three objectives, made concrete for search

Lesson 05 established the pointwise / pairwise / listwise families and their calibration properties. Here we make each concrete for ranking and work the numbers — because the difference between them is exactly the difference between "predict a number per document" and "get the order right where it matters."

Pointwise — regress or classify each document independently

Treat each (query, doc) as an independent supervised example: regress the grade (f(x) ≈ grade) with squared error, or classify relevant-vs-not with BCE. Train, then sort by predicted score. Simple, reuses all your standard regression/classification machinery, and the output can even be calibrated (05).

The structural flaw: ranking is relative, and pointwise loss is absolute. A pointwise model is penalized for predicting 2.7 when the grade is 3, even if every other document scored lower and the order is already perfect. It spends capacity matching absolute grade values that the user never sees — the user sees only the sort. Worse, it weights a query with 1000 candidates 1000× more than a query with one candidate, and it has no notion that getting rank-1 right matters more than rank-50. Pointwise is the right baseline and often a fine production choice when you also need a calibrated probability (ads — 05), but on a pure ranking metric it leaves points on the table.

Pairwise — RankNet

Stop predicting grades; predict preferences. For a pair of documents i, j under the same query where i is more relevant than j (graded higher), RankNet (Burges et al., 2005) models the probability that i should rank above j as a sigmoid of the score difference:

The loss is binary cross-entropy against the known label (the pair is correctly ordered, so the target is 1):

This is the same loss as BPR from 05 — pairwise preference, depends only on the difference s_i − s_j, so it is shift-invariant and produces a ranking score, not a probability. The point of RankNet is that it optimizes the order directly: it is only ever asked "is i above j?", never "what grade is i?".

RankNet's blind spot: every mis-ordered pair contributes the same amount of loss regardless of where in the list the inversion happens. Swapping the documents at ranks 1 and 2 (which a user definitely notices) costs the same as swapping ranks 99 and 100 (which nobody sees). The metric we actually care about — NDCG (08, 48) — is dominated by the top. RankNet doesn't know that. Listwise fixes it.

Listwise — LambdaRank / LambdaMART and the λ trick

The breakthrough (Burges, 2006/2010) is almost embarrassingly direct. RankNet's gradient on the pair (i, j) has a magnitude; call it λ_ij. LambdaRank scales that gradient by the change in the ranking metric that would result from swapping i and j in the current ranking:

The first factor is the RankNet gradient (large when the model is wrong about the pair). The second factor, |ΔNDCG_ij|, is the absolute change in NDCG if you swapped those two documents' positions, holding everything else fixed. The remarkable thing: nobody ever differentiates NDCG (it is flat-then-step, non-differentiable). LambdaRank skips the loss entirely and defines the gradients directly — the λ's — which is legal because gradient descent only ever needs gradients. The effect: the model pours its learning into the pairs whose correct ordering moves the metric most, which are overwhelmingly the pairs near the top.

LambdaMART is LambdaRank's gradient applied to gradient-boosted regression trees (GBDT / MART = Multiple Additive Regression Trees) instead of a neural net. Each boosting round fits a tree to the current λ's. This combination dominated production web-search LTR for roughly a decade, and we'll see why after the worked example.

First, NDCG itself, since the λ's are built on it. DCG (Discounted Cumulative Gain) at position p uses gain 2^grade − 1 discounted by log₂(rank + 1); NDCG normalizes by the ideal (best-possible) DCG so it lands in [0, 1].

ListNet is the other listwise style: instead of swap-deltas, it defines a probability distribution over the list via softmax of scores (the top-one probability P(i) = e^s_i / Σ_j e^s_j) and minimizes cross-entropy against the distribution implied by the true grades. Clean and fully differentiable, but it does not explicitly weight by the eval metric, and in practice LambdaMART's metric-aware gradient won the production bake-offs.

Why LambdaMART dominated — and where neural rankers fit now

For roughly 2010–2020, the default production web-search ranker was LambdaMART (GBDT trained with LambdaRank gradients — LightGBM's lambdarank objective is exactly this). Four reasons, each a real engineering property:

• Heterogeneous features, no scaling. The feature vector mixes BM25 (range 0–30), cosine (−1 to 1), PageRank (heavy-tailed), counts (integers), booleans. Trees split on thresholds, so they are invariant to monotone rescaling of any feature — no normalization, no careful feature engineering of scales. A neural net needs all of that.
• Robust on tabular data with few hundred features. GBDTs are still, in 2026, the thing to beat on medium-dimensional tabular data. With a few hundred hand-built features and tens of thousands of judged queries, a neural net usually under-performs a tuned GBDT.
• Handles missing features gracefully. Trees route missing values down a default branch — common when a feature isn't computable for some docs.
• Cheap to train and serve, easy to debug. Feature importances and split thresholds are inspectable; a PM can ask "why did this rank high" and get an answer.

Where neural / transformer rankers fit now. They win precisely where GBDTs are weak: when the signal is in raw text rather than hand-built features. A cross-encoder that reads the query and document text jointly (BERT-style, lesson 45) captures semantic match that no BM25 / proximity feature encodes — but it costs orders of magnitude more per document. So the modern stack is usually both: a fast LambdaMART (or a small neural ranker) over feature vectors ranks the few hundred candidates, and an expensive cross-encoder reranks only the top ~50. The GBDT didn't get replaced; it got a neural stage stacked on top of it (next lesson). The trees also frequently consume the neural model's score as one more feature — closing the loop on "retrieval/model scores are features."

The ranking pipeline

Read it as: retrieval scores enter on the left as two features among hundreds; the ensemble learns query-conditional thresholds on them (note the splits — "use BM25 heavily, but on a fresh-doc / navigational query weight freshness and exact title-match more"); the summed leaf values are the score; the sort is the output ranking; and the top slice flows to lesson 45's reranker.

Practical concerns

Training data from logs, and feature logging

The data engineering is where LTR projects actually fail. You need, for each impression: the feature vector exactly as the model saw it at serving time, the label (click/grade), and the position shown. The single most important rule: log the features at serving time, do not recompute them offline. If you recompute BM25 / freshness / CTR features from a later snapshot of the index, they will not match what the model scored — freshness drifts, CTR aggregates grow, the index changes. That train/serve skew silently degrades the model and is nearly invisible (the loss looks fine offline; production NDCG quietly sags). The fix is a feature-logging system that snapshots the served feature vector into the training log. This is the search analog of the calibration train/serve trap in 05 and the geometry trap in 03 — same lesson, different stage.

Position-debiased training with IPS

Recall clicks are position-biased (07). If a document at rank 1 gets clicked, part of that click is "it was relevant" and part is "it was at rank 1." Training on raw clicks teaches the ranker to reproduce the old ranker. The standard correction is Inverse Propensity Scoring (IPS): estimate the examination propensity p_r = P(user examined position r), then weight each clicked example by 1 / p_r. A click at a rarely-examined low position is strong evidence of relevance (the user had to work to see it), so it gets up-weighted; a click at rank 1 is weak evidence, down-weighted.

IPS is unbiased but high-variance (those 1/p weights blow up for tiny propensities); production systems clip the weights, smooth the propensity estimates, and lean on human judgments for the head where clicks are densest and propensity estimates least reliable.

Interactive · LTR objective explorer

A toy 5-document candidate list, each with a fixed true grade (0–4) and a current model score. Toggle the objective and watch its loss; nudge any one document's score with the slider and watch the loss, the pairwise/λ gradient on that document, and the NDCG of the resulting order all respond live. toy model

Interview prompts you should be ready for