Search, Ads & Recommender Systems

One stand-alone series, three linearized tracks over the same engine. Read any track end-to-end from first principles: the product decision, the retrieval and ranking stack, objectives and bias correction, evaluation, serving, and the failure modes a senior engineer must anticipate. Search and recommendation share roughly 90% of the machinery — the tracks cross-link that shared core instead of repeating it.

Three tracks — pick where to start

🔎 Search (lessons 40–49, ~3.5h) — the query changes everything. Query understanding → inverted index & BM25 → dense & hybrid retrieval → learning-to-rank → semantic reranking → autocomplete → personalization → relevance evaluation → system design. The newest and most self-contained track; start at 40.
🎯 Recommender systems (lessons 01–08, 12–39, ~6h) — no query, predict engagement. The retrieve→rank→rerank cascade, objectives, calibration, bias/debiasing, sequences, multimodal, evaluation, serving, data/feature engineering, and responsible constraints. The center of gravity; start at 01.
💰 Ads (lessons 09–10, 25, builds on 01–08) — turn a predicted click probability into a price. Auctions (GSP/VCG), bidding & pacing, and autobidding as multi-agent RL. A specialization on top of the recsys core; read 01–08 first.

All three share retrieval, ranking, embeddings, calibration, debiasing, and serving. Lessons 40–49 (Search) lean on 03 (ANN), 05 (losses), 07 (bias), and 08 (evaluation) rather than re-deriving them.

The mental model

First, the three product surfaces — if you're new here

This series covers three closely-related systems. A recommender system suggests content a user may like with no explicit query — YouTube Home, the TikTok feed, Amazon "you may also like" — and optimizes engagement. Search ranks results against an explicit query that states an intent the system is obligated to satisfy — web search, e-commerce search, marketplace search — and optimizes relevance. Ads rank bid-backed candidates and add an auction (lessons 09–10) that turns a predicted click probability into a price. All three share ~90% of the machinery (retrieval, ranking, embeddings, calibration, debiasing, serving); what differs is whether there is a query, whether the objective is engagement or relevance, and whether money changes hands. The Search track (40–49) teaches the query-specific half; the recsys track teaches the shared core; the ads lessons teach the auction layer.

Read each track as one chain of decisions, not a bag of model names. Every system answers the same question under different constraints: what should this user see next (given their query, if any), why, at what cost, and how do we know it helped?

product surface ──── search? ──► query understanding (normalize, spell, expand, intent) │ │ ▼ ▼ objective & labels ──► candidate generation ──► retrieval: inverted index (BM25) + ANN/dense + graph │ │ │ (search fuses lexical + dense) ▼ ▼ ▼ ranking model ◄──── features / sequences / multimodal ◄──── learning-to-rank │ ▼ rerank (cross-encoder), constraints, ads auction, diversity, exploration │ ▼ logs → debiasing → evaluation (NDCG / interleaving / A·B) → retraining │ ▼ serving, monitoring, failover, responsible guardrails

What a strong reader can do afterward

Skill	What good looks like	Lessons
Decompose the system	Given "design YouTube Home" or "design Airbnb search", draw retrieve → rank → rerank, then name the objective, labels, latency budget, and failure modes.	01, 11, 32
Build a search engine	Given "design web search" or "design product search", lay out query understanding → inverted index + dense/hybrid retrieval → learning-to-rank → semantic rerank → SERP; defend BM25 vs dense vs hybrid, size shards and the scatter-gather tail, and measure relevance with graded NDCG and interleaving.	40-49
Choose the right model family	Know when CF/MF is enough, when two-tower retrieval is required, when DLRM/DCN-style rankers fit, and when sequence, graph, multimodal, or LLM-based methods earn their cost.	02-04, 13-14, 24, 26
Reason about objectives	Translate product goals into labels, critique those labels for bias and delay, fuse multiple task heads, and calibrate probabilities when money or constrained optimization enters.	05, 07, 12, 27
Operate at scale	Explain feature freshness, train-serve skew, billion-row embedding tables, index refresh, streaming lag, data quality, canaries, rollbacks, monitoring, graceful degradation, and $/query trade-offs.	17-19, 28-30, 35-37
Evaluate honestly	Use offline metrics, replay, A/B tests, interleaving, short-video experience metrics, counterfactual estimators, causal/uplift thinking, and segment slices without confusing logged-data correlation for causal lift.	08, 27, 31, 33-34
Pass operational interviews	Answer implementation questions about A/B randomization, feature lineage, Kafka/Flink pipelines, EDA, abuse filtering, cold-start graduation, privacy, compliance, and utility trade-offs.	33-39

Track A · Search 🔎 — the query changes everything

The newest track, and the most self-contained. A query is a stated intent the system is obligated to satisfy, which changes the objective (relevance, not engagement), the candidate set (query-conditioned), and the failure modes (a hard recall floor). These ten lessons build the full search engine — query in, ranked results out — leaning on the shared retrieval/ranking/evaluation machinery from the recsys track rather than repeating it. Read 40→49 in order.

Foundations & query. What makes search different, and the first stage every engine runs on the raw query string.

Search vs. recommendation — the query changes everything

The relevance contract, query taxonomy (navigational/informational/transactional), the Zipfian query distribution, the search funnel, and how the objective and metrics differ from recsys.

Query understanding — raw string to structured intent

Normalization, tokenization, stemming, the noisy-channel spell model, query expansion (Rocchio, neural), segmentation, intent classification, NER, and attribute tagging — each as a recall/precision trade.

Retrieval. Shrink millions of documents to a candidate set — first lexically, then semantically, then both.

The inverted index & BM25 — lexical retrieval that still wins

Posting lists, compression, skip pointers, TF-IDF → the full BM25 formula with the engineering intent of k₁ (saturation) and b (length norm), BM25F, and WAND/block-max top-k pruning.

Dense & hybrid retrieval — fixing lexical's blind spot

Vocabulary mismatch, bi-encoders + ANN (builds on 02/03), learned sparse (SPLADE/doc2query), and hybrid fusion — score normalization, linear combination, and Reciprocal Rank Fusion.

Ranking & reranking. Score the candidate set with rich query-document features, then spend an expensive model on the top few.

Learning to rank for search

Query-document match features (BM25 as a feature!), judgments vs biased clicks, and pointwise/pairwise (RankNet)/listwise (LambdaMART) objectives — why GBDT LTR dominated, with worked λ math.

Multi-stage & semantic reranking

Cascade economics, bi-encoder vs cross-encoder (the precompute/accuracy trade), ColBERT late interaction, distillation, LLM rerankers, and the diversity/freshness/business blend layer.

Surfaces, evaluation & serving. The keystroke surface, the e-commerce twist, how to measure relevance honestly, and the full system.

Autocomplete, suggestions & spelling at scale

Tries/FSTs, most-popular-completion + learned ranking, the per-keystroke QPS/latency reality and head-query caching, unseen-prefix coverage, and did-you-mean spelling at scale.

Personalized & e-commerce/marketplace search

When to personalize (and when not to), multi-objective blending (relevance × conversion × margin), facets & the post-filter recall problem, and marketplace two-sidedness and seller fairness.

Search relevance evaluation

Graded judgments/qrels and pooling, every offline metric computed (P@k, MRR, MAP, NDCG, ERR), the click-bias hazard, and online interleaving vs A/B — why interleaving needs far less traffic.

Search system design & serving

The query path and the indexing path, sharding & scatter-gather tail amplification, NRT freshness, result caching, capacity napkin math, and failure modes — the Search capstone.

Track B · Recommender systems 🎯 — the shared core (Phases 1–9)

The center of gravity: no query, predict engagement. The phases below build the recommender end-to-end and contain the shared machinery the Search and Ads tracks both reuse. Track C · Ads 💰 is woven in as a specialization — lessons 09–10 (auctions, bidding & pacing) and 25 (autobidding as multi-agent RL) — and assumes Phases 1–4 first.

Phase 1 · Frame the production ranking problem

First learn the cascade. A production recommender is not "one model predicts a click"; it is a staged decision system with different latency, recall, precision, and business constraints at each stage.

The funnel — retrieve, rank, re-rank

Why production ranking is a cascade, why every stage optimizes a different proxy, and how recsys and ads share the same core machine.

System design walkthrough

The first full pass through the interview shape: requirements, cascade, objectives, latency budgets, metrics, operations, and follow-up probes.

Phase 2 · Retrieve candidates from a huge catalog

Once the surface is clear, shrink billions of items to hundreds or thousands. This phase teaches the historical path from collaborative filtering to neural retrieval, plus the embedding and index machinery that makes retrieval practical.

Candidate generation — from CF to two-tower

Collaborative filtering, matrix factorization, Funk-SVD/ALS, two-tower retrievers, multi-source recall, and why retrieval losses differ from ranking losses.

Embeddings & approximate nearest neighbour

Dot product, cosine, MIPS, HNSW, IVF, PQ, recall-latency-RAM trade-offs, sharding, updates, and when ANN is the wrong tool.

Graph neural networks for recommendation

The interaction matrix as a bipartite graph: GraphSAGE, PinSage, LightGCN, high-order connectivity, cold sparse nodes, and over-smoothing.

Phase 3 · Rank with deep models and better user understanding

Now score the candidate set with richer features. This is where sparse IDs, feature crosses, task heads, sequence behavior, and multimodal content become one ranking model.

The ranking model — LR to DLRM to DCN-v2

Sparse high-cardinality features, embedding tables, feature crosses, GBDT-to-deep evolution, DLRM/DCN/MaskNet, and what feature engineering still means.

Multi-task & multi-objective ranking

Shared-bottom, MMoE, PLE, ESMM, gradient conflict, click/watch/like/retention fusion, and why one score is never enough.

User-behavior sequence modeling

Implicit feedback, target attention, DIN/DIEN, short-term versus long-term interest, sessionization, and behavior features.

Multimodal recommendation

Text/image/video encoders, fusion strategies, CLIP-style alignment, content vectors for cold start, and the encoding cost bill.

Phase 4 · Train objectives without fooling yourself

A recommender learns from logs produced by the old recommender. This phase is the antidote to naive supervised learning: losses, calibration, sampling, position bias, off-policy correction, causal reasoning, and exploration.

Losses & calibration

Pointwise, pairwise, listwise learning-to-rank; Platt/isotonic/temperature calibration; ECE; and why ads require probabilities.

Negative sampling

In-batch negatives, sampled softmax and log-Q correction, uniform and hard negatives, false negatives, and sampling bias.

Position bias, selection bias, debiasing

Examination hypothesis, click models, IPS, doubly robust correction, intervention harvesting, and why logged data is not the world.

Cold start

User, item, and system cold start; content bridges; priors; cross-domain transfer; contextual bandits; UCB; Thompson sampling.

Reinforcement learning for recommendation

Feed as MDP, bandits as the one-step case, policy/value methods, off-policy learning from logs, and reward-design failure modes.

Causal inference & uplift modeling

Prediction versus intervention, potential outcomes, CATE, S/T/X-learners, causal forests, PSM, DID, IV, and synthetic control.

Phase 5 · Evaluate before and after launch

This phase answers "how do we know it helped?" Offline ranking metrics are useful, but production confidence comes from counterfactual care, online experiments, interleaving, segment slicing, and case-specific guardrails.

Evaluation — offline metrics & A/B tests

NDCG, MAP, AUC, log-loss, calibration error, replay, power/MDE, CUPED-style variance reduction, metric tests, bootstrap, and novelty effects.

User segmentation & profiling

User portraits, tags, RFM, K-means, embedding clusters, fast/slow profiles, segment-conditioned policy, and Simpson's paradox.

Phase 6 · Serve the system at scale

The model is only useful if it can run under an SLA with fresh features and safe rollouts. These lessons cover the engineering plane beneath the model.

Real-time & streaming recommendation

Offline, nearline, and online planes; logs and stream processors; exactly-once trade-offs; feature freshness; online learning; and latency budgets.

Deployment & serving

Export formats, serving runtimes, train-serve skew, shadow/canary/ramp/rollback, model versioning, hot updates, and monitoring.

Large-scale system optimization

Billion-row embedding tables, sharding, parameter servers, all-reduce, ANN at scale, quantization, distillation, caching, and $/query.

Feature engineering as a discipline

Categorical encoding, target leakage, numerical transforms, feature selection, streaming wrinkles, feature parity, and why embeddings do not remove craft.

Data engineering for recommendation

Batch versus stream, MapReduce/Spark, Lambda/Kappa, ETL, feature stores, point-in-time correctness, data quality, EDA, and skew/salting.

High availability & disaster recovery

Nines, error budgets, redundancy, load balancing, multi-region failover, RTO/RPO, circuit breakers, and graceful degradation.

Phase 7 · Add ads, responsibility, and frontier methods

These are specialization layers. Ads add auctions and advertiser control loops. Responsible AI adds constraints on privacy, fairness, explainability, and federated learning. Frontier lessons cover LLMs, generative recommendation, and multi-agent dynamics without pretending they replace the core stack.

The ad auction — GSP, VCG, quality score

First-price, second-price, GSP, VCG, quality score, reserve prices, ad load, externalities, and why calibrated pCTR matters.

Bidding & pacing

Autobidding, Lagrangian dual view, bid shading, daily budget pacing, control loops, and failure modes when optimizers interact.

Fairness & diversity

Filter bubbles, popularity bias, feedback loops, diversity metrics, MMR, DPP, two-sided fairness, and debiasing levers.

Explainable recommendation

Intrinsic and post-hoc explanations, feature attribution, attention caveats, templates, counterfactual explanations, trust, debugging, and compliance.

User privacy protection

Anonymization failure, k-anonymity, differential privacy, epsilon/noise/utility, DP-SGD, on-device inference, and regulation.

Federated learning for recommendation

FedAvg, communication/computation trade-offs, non-IID convergence, secure aggregation, DP, and the on-device item-table problem.

Multi-agent reinforcement learning

Non-stationarity, CTDE, MADDPG, QMIX, cooperative versus competitive settings, auto-bidders, and when MARL earns its complexity.

LLMs & generative AI in recommendation

LLM-as-judge, user simulators, semantic IDs, generative retrieval, small-set reranking, AIGC content, distillation, and the cost wall.

Phase 8 · Synthesize through cases

End by reading real systems with one reusable worksheet. The goal is to stop memorizing company names and start recognizing constraints.

Industry case studies

YouTube-style feeds, Airbnb-style marketplace search, Alibaba-style e-commerce ranking, generative recommendation, and teaching stacks read through one production worksheet.

Phase 9 · Pass the operational interview layer

The question-bank material adds a different kind of depth: not more model names, but the operational details interviewers use to tell a system builder from a paper reader. These lessons cover experiment design, short-video-specific metrics, feature lineage, streaming failures, data quality, advanced cold start, and privacy/compliance.

A/B testing deep dive

Decision metrics, guardrails, randomization units, stratified splits, MDE/power, ratio metrics, confounders, long-term holdouts, and launch rules.

Short-video metrics

Effective play, completion normalization, rewatch, fast-swipe, immersion-time density, continuous watch, retention, negative feedback, and ecosystem health.

Feature engineering deep dive

Feature lineage, entity clocks, high-cardinality encodings, dynamic normalization, time windows, user portraits, leakage, and feature importance.

Streaming pipeline ops

Kafka topic/partition design, Flink keyed state, watermarks, exactly-once semantics, backpressure diagnosis, feature-store writes, and fallback ladders.

Data quality, EDA, and anti-fraud

Schema and volume checks, missingness, outliers, long-tail distributions, bot/brush traffic, dirty-label detection, and training-data incident response.

Cold start advanced

Device/context priors, transfer learning, meta-learning, exploration budgets, graduation criteria, cold-start evaluation, and spam/security gates.

Privacy, compliance, and privacy computing

Purpose limitation, minimization, masking, anonymization limits, DP, federated learning, MPC/TEE/HE trade-offs, deletion, transfer, and utility measurement.

Goal	Read this path
Core ranking interview	01 → 02 → 03 → 04 → 05 → 06 → 07 → 08 → 11 → 32 → 33
Recsys production depth	01-08 → 12-19 → 28-32 → 35-37
Short-video / feed systems	01-08 → 12 → 13 → 15 → 16 → 17 → 20 → 31 → 32 → 34 → 38
Ads teams	01-10 → 12 → 17-19 → 25 → 27 → 30 → 32 → 33 → 39
Frontier / research-flavored	13-16 → 20-27 → 31-32 → 38-39
Question-bank cram	33 → 34 → 35 → 36 → 37 → 38 → 39, after reading 01-08
Search systems	40 → 41 → 42 → 43 → 44 → 45 → 46 → 47 → 48 → 49 (self-contained; 01 helps but isn't required)