Dense & hybrid retrieval — fixing lexical's blind spot

BM25 matches tokens; it cannot see that "flat" and "punctured tyre" are the same intent. Dense retrieval matches meaning but fumbles the exact part number BM25 nails. Production search is neither — it is the fusion of both, and the fusion is the part worth getting right.

Lexical's blind spot, made concrete

Lesson 42 built BM25 over an inverted index: a query term scores a document by how often the term appears (term frequency), discounted by how common the term is across the corpus (inverse document frequency), normalised by document length. It is fast, exact, and unreasonably strong. But re-read what it actually scores: shared tokens. If the query and the relevant document share no tokens, BM25 returns a score of zero — not "low", zero — because every term in the query has term-frequency 0 in the document, so every per-term contribution is 0.

Here is the failure with numbers you can check by hand. Query: "how to fix a flat". Relevant document: "repairing a punctured tyre". After stop-word removal the query content terms are {fix, flat} and the document content terms are {repairing, punctured, tyre}. The intersection is empty. BM25 is a sum over query terms of (idf × saturated tf); with tf = 0 for both fix and flat in this document, the score is 0 · idf(fix) + 0 · idf(flat) = 0. A perfect-intent match scores identically to a document about quantum chromodynamics. This is vocabulary mismatch: the same meaning expressed in disjoint words. Synonyms (fix/repair), morphology (tyre/tire), paraphrase (flat / punctured tyre), and abbreviation all trigger it. Studies going back to Furnas et al. (1987) found two people pick the same word for a concept under ~20% of the time — vocabulary mismatch is the common case, not the edge case.

Classical search papered over this with hand-built tricks: stemming (tyre→tyr), synonym expansion dictionaries (fix→{repair, mend}), and spelling correction. Each helps a little and each is brittle — a curated synonym list never has "flat ↔ punctured tyre", and adding loose synonyms hurts precision. The structural fix is to match on a representation of meaning, which is what dense retrieval does.

Dense bi-encoders for search

The machinery is the two-tower model of lesson 2, re-pointed at text. One tower encodes the query, the other encodes the document (or, as we will see, a passage); both emit a vector in the same d-dimensional space; relevance is their dot product or cosine. At index time you encode every document once and load the vectors into an ANN index. At query time you encode the query once and run an ANN lookup — HNSW or IVF-PQ, exactly the index families of lesson 3. We do not re-derive any of that here; the geometry (dot vs cosine vs L2), the MIPS-is-not-a-metric subtlety, and the recall/latency/RAM triangle are all lesson 3's job. Dense retrieval for search is "two-tower + ANN with text towers." What is genuinely new is everything below.

The DPR recipe — contrastive training with hard negatives

DPR (Dense Passage Retrieval, Karpukhin et al. 2020) is the canonical recipe and the one to be able to sketch on a whiteboard. Encode query q with a BERT-style encoder E_Q (take the [CLS] vector), encode passage p with a separate encoder E_P; score is s(q,p) = ⟨E_Q(q), E_P(p)⟩. Train with a contrastive loss: for a batch of (query, positive-passage) pairs, push the positive's score up and every other passage's score down. This is precisely the in-batch-negatives softmax from lesson 2:

L = − Σ_i log [ exp(s(q_i, p_i⁺) / τ) / ( exp(s(q_i, p_i⁺)/τ) + Σ_j exp(s(q_i, p_j⁻)/τ) ) ]

The negatives p_j⁻ are the other passages in the batch (free) plus mined hard negatives — and the hard negatives are what separate a strong retriever from a mediocre one. The DPR paper's single most cited finding: adding one BM25 hard negative per query (a passage that BM25 ranks high but that does not answer the query) is worth more than dozens of random negatives. Why? In-batch negatives are almost all trivially easy — a random passage from another query is obviously irrelevant, so it produces a near-zero gradient. The model learns the easy boundary fast and then stops improving on the boundary that matters: distinguishing a passage that looks relevant (shares words, shares topic) from one that is relevant. Hard negatives live exactly on that boundary. This is the search-flavoured instance of the negative-sampling story built in lesson 6 — the same logQ correction and mixed-negative reasoning applies; here the "hard" miner of choice is usually BM25 itself, which is a nice symmetry (lexical retrieval generates the hard cases that teach the dense model).

Query–document asymmetry and why search chunks

Two search-specific design facts that recsys two-tower glosses over:

• Asymmetry. A query is short (2–5 words, often an underspecified fragment); a document is long and well-formed. Encoding both with the same tower wastes capacity and slightly hurts — the optimal representation of "how to fix a flat" is not produced by the same function that should represent a 600-word repair guide. DPR uses two separate encoders for exactly this reason. (Many production systems share weights to halve memory and accept the small quality cost — a real trade-off, not a free lunch.)
• Passage-level encoding + chunking. A single vector cannot fluently summarise a 5,000-word document; the encoder has a hard token limit (e.g. 512), and even within it, one mean-pooled vector blurs a document that covers ten subtopics. The fix is to chunk the document into passages (a few hundred tokens each, often overlapping), encode each chunk to its own vector, and index the chunks. A query then retrieves passages; you map back to the parent document and dedupe. This is why dense search is "Dense Passage Retrieval" and not "Dense Document Retrieval" — granularity is a first-class decision. Chunk too coarsely and you reintroduce the blur; chunk too finely and you explode the index and lose the context a passage needs to be interpretable.

A late-interaction variant, ColBERT, keeps a vector per token instead of one per passage and scores via a sum-of-max-similarities ("MaxSim") — far more expressive, far more storage. That is lesson 45's territory (it sits between bi-encoder retrieval and cross-encoder reranking); here we stay with single-vector bi-encoders, which are what feed the ANN index.

Learned sparse retrieval — the bridge from 42 to dense

Before fusing two indexes, there is a clever middle path: keep the inverted index of lesson 42 but make a model decide which terms appear and with what weight. You get semantic recall and lexical efficiency and exact-match — all inside the data structure you already operate.

Position this clearly in interviews: learned sparse is the bridge between lexical and dense. It has dense's superpower (semantic expansion, learned weights) but lives in lexical's home (the inverted index), so it inherits exact-match on rare tokens and the operational maturity of lesson 42's infrastructure — no second index, no second serving system, no fusion needed in principle. The catch: the expansion is bounded by the vocabulary and the model's context window, it can over-expand (precision loss when it injects loosely related terms), and posting lists get longer (the L1 penalty trades quality for index size). Many teams run BM25 or SPLADE as the lexical branch and still fuse with a dense branch, because dense's continuous space catches paraphrase that even SPLADE's term-level expansion misses.

Hybrid fusion — the core of the lesson

You now have (at least) two ranked lists for a query: one from the lexical branch (BM25 / SPLADE over the inverted index) and one from the dense branch (bi-encoder over the ANN index). Each is strong where the other is weak. Fusion combines them into one candidate set for the ranker. The naïve idea — "just add the scores" — is where careers go to learn humility.

Why you cannot add raw scores

BM25 scores are unbounded sums of idf terms; a query with rare terms can produce a BM25 score of 25, a query with common terms a score of 4. Dense cosine scores live in [−1, 1]. Adding s_lex + s_dense means BM25 utterly dominates whenever its scale happens to be large — fusion silently degenerates into "BM25 with a rounding error." The scales are incomparable, and worse, BM25's scale varies per query (it depends on the idf of the query terms), so there is no single constant you can divide by. The train/serve pitfall: you tune a fusion weight on one query distribution, the query mix shifts, BM25's typical magnitude shifts with it, and your carefully chosen weight is now wrong with no error anywhere. Two families of fix:

Family A — normalise, then linearly combine

Map each list's scores to a common range, then take a weighted sum. Min-max normalisation per list per query: s' = (s − min) / (max − min), giving each list a [0,1] range. Then

s_fused(d) = α · s'_lex(d) + (1 − α) · s'_dense(d)

with α ∈ [0,1] tuned on a labelled set. (Z-score normalisation — s' = (s − μ)/σ per list — is the alternative; it centres and scales by spread rather than range.) This is intuitive and lets you dial the lexical/dense balance. Its weakness is exactly the thing min-max is sensitive to: an outlier score blows up the denominator. If one document gets a freak-high BM25 score (a query term that is astronomically rare), then max − min is huge, every other document's normalised score is squashed toward 0, and the lexical branch effectively contributes nothing for that query. We will watch this happen with numbers below.

Family B — fuse on ranks, not scores: Reciprocal Rank Fusion

Reciprocal Rank Fusion (RRF, Cormack et al. 2009) throws away the scores entirely and uses only each document's rank in each list:

RRF(d) = Σ_{lists ℓ} 1 / ( k + rank_ℓ(d) )

where rank_ℓ(d) is d's 1-based position in list ℓ (rank 1 = best), and k is a smoothing constant, conventionally k ≈ 60. A document absent from a list contributes 0 from that list. Because it consumes only ranks, RRF is completely immune to the score-scale problem — BM25's magnitude, dense's [−1,1], a freak outlier: none of it matters, because rank 1 is rank 1 regardless of whether the top score was 25 or 0.9. This robustness is why RRF is the default fusion in Elasticsearch, OpenSearch, and Vespa hybrid search. The constant k controls how sharply the top ranks dominate: small k (say 10) makes rank-1 worth far more than rank-5; large k (say 100) flattens the curve so deep-but-agreed-upon documents can outscore a single list's rank-1. k = 60 is the empirically robust middle the original paper recommends.

Worked numbers — RRF on two ranked lists

Six documents, A–F. The lexical (BM25) branch and the dense branch each rank the ones they found. Note: E was found only by dense (BM25 score 0 — vocabulary mismatch), and F only by lexical (a rare SKU the dense encoder fumbled). C is ranked mediocre by both — neither puts it top — but both agree it's relevant.

Sorting by RRF total gives the fused ranking: A (0.032266) > D (0.032018) > C (0.032002) > B (0.031514) > E (0.015625) > F (0.015385). Three things to read off this, each a teaching point:

• Agreement is rewarded. C is rank 3 in lexical and rank 2 in dense — top of neither list — yet RRF promotes it to fused rank 3, ahead of B which was lexical rank 2. Why: C appears respectably in both lists, accumulating two contributions, while B is strong lexically but weak in dense (rank 5). RRF rewards consensus across methods, which is exactly the behaviour you want — a document both branches like is a safer bet than one only the lexical branch likes. This is the "doc ranked low-ish by both that fusion promotes" the brief asks for: C never wins a single list but lands at #3 fused.
• Single-method discoveries survive. E (dense-only) and F (lexical-only) each get only one contribution, so they land below the consensus documents — but they are not lost. E is the "punctured tyre" recall save that BM25 scored 0; F is the "MX-4" SKU that dense fumbled. Fusion keeps both in the candidate set for the ranker to judge, which is the whole point: neither branch alone surfaces both.
• D, dense's rank-1, does not automatically win. It is fused rank 2 because lexical only had it at rank 4. RRF does not let one list's confidence override the other's skepticism.

Worked numbers — min-max linear fusion, and how an outlier wrecks it

Now the same documents under min-max + linear fusion, to feel the contrast. We need raw scores. Suppose BM25 raw scores are A=12.0, B=9.0, C=6.0, D=4.0, F=2.0 (E absent → 0), and dense cosines are D=0.81, C=0.78, A=0.74, E=0.70, B=0.55 (F absent → 0). Min-max each (per list, over present docs):

Lexical range [2.0, 12.0], span 10.0: A=1.00, B=0.70, C=0.40, D=0.20, F=0.00. Dense range [0.55, 0.81], span 0.26: D=1.00, C=0.885, A=0.731, E=0.577, B=0.00. With α = 0.5:

Fused order: A > C > D > B > E > F — broadly sane, fairly close to RRF. Now inject the outlier. A new query term is astronomically rare, so document A's BM25 score jumps to 120 (idf is a log of N/df; one ultra-rare term can dominate). The lexical range becomes [2.0, 120.0], span 118: now A=1.00, B=0.059, C=0.034, D=0.017, F=0.00. The whole lexical branch except A has been crushed toward zero by the blown-up denominator. Re-fuse with α=0.5:

B, which was fused rank 4, has collapsed to rank 5 — and more importantly the entire lexical signal for B, C, D has been wiped out: their fused scores are now driven almost entirely by the dense branch, because min-max squashed their lexical contributions to near zero. One outlier in one list silently turned a hybrid system into a near-dense-only system. RRF is immune to this: A's BM25 score jumping from 12 to 120 does not change its rank (still rank 1), so the RRF table above is completely unaffected. That immunity — not some accuracy edge — is the practical reason RRF is the production default. (Z-score has the same outlier disease: a single huge value inflates σ and shifts μ, distorting every normalised score.)

Learned fusion

The grown-up version: stop hand-tuning α and k, and learn the combination. Feed the per-document signals — BM25 score, dense score, both ranks, query length, query-has-rare-token, etc. — into a small model (logistic regression or GBDT) trained on relevance labels to predict the fused score. This is conceptually learning to rank with fusion features, which is exactly where lesson 44 picks up. The trade-off: more power and per-query adaptivity (it can learn "trust lexical more when the query has a rare token"), at the cost of needing labelled data, a training pipeline, and the usual train/serve discipline. Most teams start with RRF (zero training, robust), add learned fusion once they have judgments and a measurable gap.

The hybrid architecture

Two operational notes the diagram encodes. First, fusion happens before the ranker: it builds the candidate set the ranker scores, so a document only one branch found still gets a fair hearing from the ranker. Second, the dense branch retrieves passages but the candidate set is parent documents — you dedupe chunks back to their document before (or during) fusion, usually keeping the document's best-scoring chunk.

Interactive · hybrid fusion explorer

Two fixed ranked lists over a shared 6-document set — a lexical ranking and a dense ranking. One document, ★ ideal, is the truly-relevant answer; it is ranked mediocre by both branches (the classic "consensus but never #1" case) so a good fusion method should still float it up. Pick a fusion method and dial its knob; watch where the ideal document lands. The arithmetic is the real RRF / min-max formula above — no approximation.

When NOT to bother with dense retrieval

Dense and hybrid are not free — a second model, a second index, a second serving path, GPU/encoder cost on the query path, and fusion to tune and monitor. Reach for them by argument, not reflex. Skip or defer dense when:

• Small or structured corpus. A few thousand documents, or records with clean structured fields (SKU, category, price). BM25 plus filters is exact, debuggable, and instant; the vocabulary-mismatch problem barely bites at small scale because you can curate synonyms. Dense adds infrastructure for marginal recall.
• Navigational / known-item queries. The user typed an exact product name, an error code, a username, a citation. These are lexical by nature — the user knows the string. Dense retrieval's generalisation is a liability here (the MX-4 problem), and BM25 already returns the exact item at rank 1. Many real query streams are dominated by navigational intent; measure your mix before assuming you need semantics.
• Latency or cost budget is tight. The dense branch adds a query-encoder forward pass (often a transformer) plus an ANN lookup on the hot path, and the encoder may need a GPU. If you are at single-digit-millisecond budgets on CPU-only infra, that may not fit. Learned sparse (SPLADE/doc2query) is the compromise — semantic recall with no second index and no query-time encoder for doc2query.
• No labelled data and no eval. You cannot tune α, choose k, validate hard-negative mining, or even know dense is helping without judgments (lesson 48). Shipping dense blind can lower precision (it injects plausible-but-wrong semantic neighbours). Start with BM25, build the eval set, then add dense and prove the lift.

Interview prompts you should be ready for