The ranking model — LR to DLRM to DCN-v2
Retrieval handed you K₁ candidates. The ranker decides which dozen a human sees. Fifteen years of architecture history, compressed.
What the ranker actually eats
The shaping constraint: a production ranker takes (user, item, context) and must emit a calibrated score in O(100 µs). Inside the triple, features come in four shapes, each handled by a different sub-network. Confusing the shapes is the most common junior error.
| Feature kind | Examples | Cardinality / shape | Handled by |
|---|---|---|---|
| Sparse categorical | user_id, item_id, ad_id, query token, geo, device_model | 10⁶ – 10⁹ unique values, one ID per slot | Embedding table lookup. One row per ID (often after hashing). |
| Sparse multi-valued | last 100 clicked item_ids, queries in session, watched videos | Variable-length list of IDs per row | Lookup each ID, then pool (sum/mean) or attend over history — target-attention (DIN), GRU + interest-evolution (DIEN), or hard-retrieve-then-attend for thousand-length histories (SIM). Derived in lesson 13. |
| Dense numerical | item age (days), price, ctr_30d, user account age | Scalars in ℝ | Standardize / bucket / log-transform, then feed an MLP. |
| Cross / context | query × ad cosine, time-of-day, hour-of-week, slot position | Mostly derived at scoring time | Either pre-computed feature, or implicit via the cross-network. |
An interviewer who asks "how do you featurize a user" is asking whether you know those four shapes exist and enter the network through different doors.
The embedding table is the model
A hashed user_id space of 10⁸ at embedding dim 64 is 6.4 × 10⁹ parameters. In one table. A production ranker has tens of such tables. The MLP on top is maybe a million parameters — six orders of magnitude smaller.
This drives the production-systems story. The MLP fits on one GPU; the embeddings don't fit on a rack. Hence: hashing tricks, model-parallel tables, sharded parameter servers, hot-row caching — covered in the system_ml folder. For this lesson, the imbalance is the headline fact.
The architecture timeline
Logistic regression (~2010)
One scalar weight per feature, one bias, sigmoid: p(click | x) = σ(w · x + b). "Features" are one-hot encodings of every sparse ID and bucketed dense values. Interactions — e.g., the (query="running shoes", ad_id=42) pair being high-CTR even though query and ad in isolation aren't — must be hand-engineered: query_token × ad_id becomes a new sparse field. The number of these crosses explodes combinatorially.
LR is interpretable, trivially parallelizable (FTRL, online learning), and almost free to serve. It defined the "feature-engineering era" of Big Tech ads ML from roughly 2008–2015 — engineers writing cross-feature recipes.
Factorization Machines (Rendle 2010) and FFM (Juan 2016)
The first move toward learned crosses. FM keeps the linear term, then adds explicit pairwise interactions via latent vectors:
ŷ(x) = w₀ + Σᵢ wᵢ xᵢ + Σᵢ Σⱼ>ᵢ ⟨vᵢ, vⱼ⟩ xᵢ xⱼ
Each feature gets a k-dim latent; their interaction is the dot product. You don't enumerate O(d²) crosses — you learn them in O(d k) parameters. FFM gives each feature a different latent per "field" (interactions can differ across query vs ad). FM/FFM dominated the Criteo/Avazu Kaggle era.
Wide & Deep (Cheng et al. 2016, Google Play)
The template every subsequent architecture inherits: a wide LR head plus a deep MLP head, summed at logit. p = σ(w_wide · x_wide + MLP_deep(x_deep) + b).
- Wide handles memorization: hand-crafted crosses, exact (user, item) co-occurrence.
- Deep handles generalization: embeddings → MLP, so similar users see similar items even without co-occurrence.
This decomposition — explicit crosses for memorization + neural net for generalization — is the pattern DLRM and DCN both inherit. The idea from Wide & Deep that survived is the hybrid itself.
DeepFM (Guo et al. 2017, Huawei) — the FM→DNN bridge
Wide & Deep left one chore on the table: the wide side is still hand-crossed. Someone has to write query_token × ad_id, user_city × item_category, and the rest — the exact LR-era recipe-writing the deep era was supposed to retire. DeepFM removes that chore. It replaces the wide LR head with an FM head that learns the second-order crosses, and — the load-bearing trick — the FM head and the deep MLP share one embedding table:
ŷ = σ( y_FM + MLP_deep(e) ), where both y_FM and the MLP read the SAME embeddings e
Why the shared embedding matters: in Wide & Deep the wide and deep sides own separate parameters, so a rare cross seen only on the wide side gets no help from the deep side's representation. In DeepFM the FM term and the MLP train the same vᵢ jointly — the order-2 signal and the deep generalization signal reinforce one embedding instead of splitting the gradient. And there is no feature-engineering pipeline feeding the wide side: you hand DeepFM raw sparse IDs and it learns the crosses end-to-end. That is the whole pitch in one line — Wide & Deep generalizes but still needs hand-crafted crosses; DeepFM learns the order-2 crosses and shares embeddings with the deep side, so manual cross engineering drops to zero.
The FM head is the same pairwise term from the FM section, computed efficiently. Naively it is Σᵢ Σⱼ>ᵢ ⟨vᵢ, vⱼ⟩ xᵢ xⱼ — O(d²) over d active features. The standard identity collapses it to two linear scans per latent dimension:
Σᵢ Σⱼ>ᵢ ⟨vᵢ, vⱼ⟩ xᵢ xⱼ = ½ Σ_f [ (Σᵢ v_{i,f} xᵢ)² − Σᵢ v_{i,f}² xᵢ² ]
Naive double loop (3 dot products): ⟨v₁,v₂⟩ + ⟨v₁,v₃⟩ + ⟨v₂,v₃⟩ = 2·3 + 2·1 + 3·1 = 6 + 2 + 3 = 11.
Identity (two scans): square-of-sum = (2+3+1)² = 6² = 36; sum-of-squares = 2² + 3² + 1² = 4+9+1 = 14; so ½(36 − 14) = ½·22 = 11. ✓ Same answer.
The point is the cost. With d active features and latent dim k, the double loop is \binom{d}{2}·k multiply-adds; the identity is O(d·k) — one pass to accumulate Σ vᵢ, one for Σ vᵢ². At d = 100 active features, k = 16: naive ≈ 4,950 × 16 ≈ 79K ops; identity ≈ 100 × 16 × 2 ≈ 3.2K ops — a ~25× cut that comes free from algebra, which is exactly why FM-style order-2 is cheap enough to leave switched on under a 100 µs budget.
The lineage then forks. DeepFM's FM head gives you order-2 crosses for free but stops there; higher-order interactions are left to the deep MLP to discover implicitly. DCN's CrossNet (next-but-one) is the explicit answer to "what replaces the hand-crossed wide side when you also want order-3 and up": instead of an FM that tops out at order-2 or a wide head you hand-cross, a stack of L cross layers builds learned interactions up to order L+1. Read the architecture timeline as a single thread — where do feature crosses come from? LR: you write them. Wide & Deep: still write the wide ones. FM/DeepFM: the FM head learns order-2 from shared embeddings. DCN-v2: CrossNet learns order ≤ L+1 and the hand-crossed wide side disappears entirely.
DLRM (Naumov et al. 2019, Meta)
The production-canonical Meta-style ranker. Forward pass:
Note: the dense bottom MLP's output is treated as an additional "embedding" that participates in the pairwise dot products alongside the sparse-feature embeddings; it ALSO skip-connects directly to the top MLP.
Three pieces: bottom MLP projects dense features to embedding dim; pairwise dot products ⟨eᵢ, eⱼ⟩ compute explicit second-order interactions (the deep analogue of FM's pairwise term); top MLP eats the dot-product vector + dense bottom output and emits the logit.
DLRM's architectural commitment is that second-order interactions matter and should be computed explicitly. The top MLP can still learn higher-order effects, but order-2 is given for free.
DCN / DCN-v2 (Wang et al. 2017 / 2021)
DCN replaces "all pairwise dots" with a stack of "cross layers" that learn higher-order interactions efficiently. The DCN-v2 cross layer:
x_{l+1} = x₀ ⊙ (W_l · x_l + b_l) + x_l
Each layer multiplies x₀ elementwise with a learned linear projection of x_l, then adds the residual. Stacking L layers expresses interactions up to order L+1 — with only L matrices of parameters, not O(d^{L+1}). DCN-v1 used a rank-1 vector form; v2's matrix form is more expressive and what production deploys.
DCN-v2 is usually paired with a parallel deep MLP — again echoing Wide & Deep: an explicit crosses tower next to an implicit MLP tower.
user_id × item_category × time-of-day — a specific user buys snacks, but only in the evening. DLRM computes pairwise dots, so it sees (user × category), (user × time), (category × time) — three order-2 terms — but never the joint triple. It can only capture the three-way effect if you hand-engineer a new sparse feature category×time (back to the LR-era cross-writing it was supposed to kill). DCN-v2 with L = 2 cross layers expresses interactions up to order 3 automatically: layer 1 builds order-2, layer 2 multiplies by x₀ again to reach order-3. The trade-off in one line: DCN-v2 buys higher-order expressiveness with a cheap stack of L matrices, at the cost of being harder to debug than DLRM's interpretable "these two features interacted." That is the whole DCN-vs-DLRM debate in a sentence.
Why GBDTs lost
From roughly 2010–2016, gradient-boosted trees (XGBoost, LightGBM) were the strongest off-the-shelf CTR model on tabular data. Then they stopped winning, and the reason is structural, not fashion-driven.
| GBDT | Deep ranker (DLRM / DCN) | |
|---|---|---|
| Sparse high-cardinality features | Painful — can't split on 10⁹ user_ids; you bucket or target-encode and lose information. | Native — embedding tables give every ID a learned vector. |
| Parameter sharing across users | None — every leaf is its own decision. | Two users with similar histories share gradient signal through shared embeddings. |
| Multi-task heads | One tree per task; embeddings not shared. | Shared trunk + per-task heads; sample-efficient (MMoE / PLE — lesson 12). |
| Sequence features (user history) | Flatten or hand-aggregate. | Pool / attend natively (DIN / DIEN / SIM — lesson 13). |
| Strengths that remain | Small dense tabular problems, fast iteration, no GPU. | Anything with sparse-categorical + multi-task + sequence. |
The transitional architecture was Facebook's 2014 "Practical Lessons" paper: a GBDT extracts leaf-index features that feed an LR head. It was production for a few years. Once embedding tables made sparse IDs first-class, the GBDT step became dead weight.
What "feature engineering" means now
Feature engineering didn't go away with deep learning — it moved. You no longer write query_token × ad_id by hand; the network learns it (that is exactly what DeepFM's FM head and DCN's CrossNet automate). But you do decide the following — and the full toolkit (time-decay half-lives, sliding windows, Bayesian smoothing of sparse rates) is lesson 28:
- Vocabulary and hash sizes. A 10⁸ user_id space hashed into 10⁶ buckets has collisions on every popular user.
- Embedding dimensions per field. Geo with 200 values wants dim 8; user_id with 10⁹ values can carry dim 128. Uniform dims waste parameters. A common heuristic is dim ≈ k·log₂(cardinality) (or a small power of the cardinality), capped at ~128: geo (200 values) → log₂ 200 ≈ 8; query_token (10⁶) → ~20; user_id (10⁹) → ~30, capped. The logic: a field with few values can't carry more than a few bits of identity, so a high dim is just unused capacity, while a billion-value field needs room to keep its IDs distinguishable.
- Which user history to surface. Clicks? Impressions? Watched-to-completion? Each is a different multi-valued feature.
- Bucketing dense features. Raw
priceis fine;log(1+price)in 32 quantiles is usually better — MLPs learn monotonic responses faster from buckets. - Position as a side input. Position is a feature at training time (it causes clicks). It is not a feature at inference (you're deciding it). Lesson 7.
- Defining the label. Most consequential of all — lesson 5. The per-head loss shapes (BCE for click, Huber/Gamma for watch-time) and how several heads fuse into one score live in lesson 5 and lesson 12.
Trade-off table
| Capacity | Train cost | Inference cost | Interpretability | Feature-eng load | |
|---|---|---|---|---|---|
| LR | Linear only | Trivial | Trivial | High — per-feature weights | Very high — all crosses by hand |
| FM / FFM | Linear + 2nd order | Low | Low | Medium | Medium |
| Wide & Deep | LR + MLP | Medium | Medium | Low | Medium — wide side still hand-crossed |
| DeepFM | FM 2nd order + MLP (shared emb) | Medium | Low — FM cross via the identity | Low | None — FM learns order-2, no hand crosses |
| DLRM | Explicit 2nd order + MLP | High (sparse-heavy) | High — pairwise dots are O(K²) | Low | Low — model learns crosses |
| DCN-v2 | Learned order ≤ L+1 | High | Medium — L cross layers | Low | Low |
DCN-v2 vs DLRM is the live debate in current production. DCN-v2 often matches or modestly beats DLRM on the same feature set; gains are dataset-dependent and many production teams report parity. DLRM is older, simpler, easier to scale embedding-wise. Many teams run both — DCN cross-stack and DLRM dots, concatenated before the top MLP.
Interactive · feature-importance composer
Toggle features. The widget shows an expressiveness bar and a heuristic ΔAUC vs LR (toy — directionally correct, not literal). The diagnosis calls out the failure mode of each combination.
Interview prompts you should be ready for
- "Walk me through DLRM's forward pass." (Sparse → embeddings; dense → bottom MLP; all pairwise dot products; concat with dense bottom; top MLP → logit. Name the three sub-networks.)
- "Why do production rankers have an embedding table with billions of parameters but an MLP with maybe a million?" (Capacity-per-parameter argument: each sparse ID needs its own row to be distinguishable; the MLP is shared across all (user, item) pairs and gets reused on every example.)
- "Why use DCN-v2 instead of DLRM?" (Probes: DLRM hardcodes order-2 interactions via dot products; DCN-v2 cross layers learn order ≤ L+1 with a stack of L matrices. If higher-order effects matter — they usually do — DCN-v2 has the capacity to express them and DLRM doesn't.)
- "Your model has 6B parameters and 99% of them are in one user_id table. How do you train this on 8 GPUs?" (Embedding model-parallelism: shard the table by ID across GPUs, all-to-all the looked-up rows, then data-parallel the dense top. This is the canonical "torchrec" / FBGEMM / "DLRM-style" answer.)
- "Embed all features into 64-dim, or use per-feature dim? Trade-offs?" (Uniform dim is simpler and lets you concat freely; per-feature dim saves parameters on low-cardinality fields and lets high-cardinality fields breathe. Production systems typically per-feature.)
- "When would you keep an LR baseline alive in 2025?" (Three reasons: as a sanity-check during ranker rollouts; for cold-start segments where the deep model has no signal; for explainability in policy/legal contexts. Plus: LR is what you fall back to during a deep-model serving outage.)
- "You add 100 new sparse features to a DLRM. AUC barely moves. Why?" (Several real causes: the embedding tables are under-trained for the new fields, hashing collisions, embedding dim too small, pairwise dots now have 100 noisy terms swamping the signal, or the features are correlated with existing ones.)
- "DeepFM vs Wide & Deep — what specifically does DeepFM remove, and what does the shared embedding buy?" (Removes the hand-crafted wide crosses: the FM head learns order-2 interactions from raw sparse IDs, so there is no cross-feature engineering pipeline. The shared embedding means the FM term and the deep MLP train one set of latents jointly, so the order-2 and generalization signals reinforce the same vectors instead of splitting gradient across the separate wide/deep parameter sets Wide & Deep uses. Trade-off: DeepFM's explicit crosses stop at order-2; for higher order you reach for DCN's CrossNet.)
- "The FM second-order term is a sum over all feature pairs — that's O(d²). How is it cheap enough to serve?" (The identity ½Σ_f[(Σvᵢxᵢ)² − Σvᵢ²xᵢ²] collapses the double sum to two linear scans per latent dim, giving O(d·k) instead of O(d²·k). Concretely at d=100, k=16 that's ~3.2K ops vs ~79K — a ~25× cut from algebra alone, which is why FM-style order-2 stays on under a tight serving budget. Bonus signal: it's the same trick that makes DLRM's explicit dots the expensive part, since DLRM does keep the O(K²) pairwise stage.)