Case study — a recommendation ranking service

The previous cases were GenAI systems. This one is the classic production ML shape: retrieve candidates, hydrate features, rank, rerank, diversify, log, and learn from biased feedback. The model is not the bottleneck by itself. The wall is a three-way trade among candidate recall, feature freshness, and p99 latency.

The brief

Design a home-feed ranking service. Inventory: 100M items. Traffic: 50k feed requests/s at peak. SLO: p99 < 120 ms end-to-end. Each request returns 20 items. The product cares about long-term retention, not only clicks. New items and new users must get exposure. Design the system and the feedback loop.

Beginner translation

A recommender is a decision system that answers: "which 20 items should this user see next?" A candidate is an item that might be shown. A ranker scores candidates. A reranker applies product rules like diversity, safety, freshness, and exploration. The hard part is not scoring one item; it is reducing 100M possible items to 20 good ones in 120 ms without trapping the system in yesterday's feedback loop.

Before reading: what binds first? Not GPU memory. Not TTFT. The first wall is you cannot score 100M items per request. Even if scoring one item took only 0.01 ms, scoring 100M would take about 1,000 seconds for one feed. You must build a cascade and preserve enough recall that the expensive ranker ever sees the good items.

Stage 0 · Requirements → budgets

Start with the synchronous p99 budget. The user waits for the whole chain, so every stage spends from the same 120 ms. A plausible split:

Stage	p99 budget	Output
Request/auth/context	10 ms	User/session context
Candidate retrieval	30 ms	~1000 candidates
Feature hydration	25 ms	User/item/context features
Ranker	30 ms	Top ~100
Rerank/diversify/policy	15 ms	20 final items
Serialize/log	10 ms	Response + impression log

These numbers are negotiable, but the discipline is not: every stage spends from one latency budget. If retrieval expands from 1000 to 5000 candidates, the ranker and feature store must pay for it.

How to think before designing

The linear argument is: 100M items is impossible to score directly, so retrieve a smaller set; retrieval might miss good items, so measure recall; richer ranking needs features, so budget feature freshness and lookup p99; clicks are biased by what you showed, so log impressions and explore; the product cares about retention, so clicks are a proxy, not the final objective.

Peak arithmetic

At 50k requests/s and 1000 retrieved candidates/request, the ranker sees 50k · 1000 = 50M candidate scores per second. If each candidate needs 200 feature values, naive per-candidate lookups would imply 10B feature reads/s, which is absurd. Therefore features must be batched, cached, denormalized, or prejoined; the ranker must be cheap enough for tens of millions of scores/s; and candidate count is a capacity knob, not just a quality knob.

Stage 1 · The cascade

Retrieval uses a two-tower model: one small model turns the user/session into an embedding, another turns each item into an embedding, and approximate nearest-neighbor search finds nearby items quickly without scanning all 100M. Ranking hydrates richer features and scores ~1000 candidates with a tree model or deep ranker. Reranking applies diversity, freshness, safety, dedupe, business constraints, and exploration.

score(u,i) = <g(user/session), h(item)>

The two-tower training loop usually uses sampled softmax or in-batch negatives: make the clicked/watched item score higher than sampled non-clicked items. The item tower h(item) is precomputed and loaded into the ANN index; the user tower g(user) runs online per request. That creates an operational contract: item embeddings and the ANN index need a refresh cadence. Hot items may refresh in minutes, the full index may rebuild hourly/daily, and every ranking log should record the index version so offline eval knows which candidate universe was actually available.

Binding constraint, stage 1

Candidate recall under latency. If retrieval misses the best item, no ranker can recover it. Approximate nearest-neighbor indexes expose knobs that trade recall, latency, and RAM: search more of the graph, scan more clusters, or compress vectors more aggressively. The first design question is not "which ranker?" It is "how many good candidates can I retrieve inside 30 ms?"

Stage 2 · Feature freshness and hydration

The ranker wants user history, item stats, creator quality, session context, freshness, safety features, and cross features. Some are batch, some nearline, some online:

Feature plane	Examples	Update cadence	Serving strategy
Offline	Item embedding, creator quality, long-term user interests	Hours-days	Precompute and store
Nearline	Recent clicks, impressions, hides, follows	Seconds-minutes	Stream aggregates into online store
Online/session	Current page/session intent, request context	Milliseconds	Pass in request or session cache

Feature hydration can dominate p99. Batch item features by candidate ID; co-locate hot features; prejoin item metadata into the index where possible; define fallbacks. Monitor feature freshness and missing rates as production SLOs, not data-team afterthoughts.

Point-in-time correctness

A feature is legal for training only if feature_ts ≤ impression_ts. Anything computed after the impression is a label or future leak, not a serving-time feature. Replaying online requests through the offline feature pipeline should track skew_rate = count(|online − offline_replay| > tol) / count_checked. A low-latency ranker trained on leaked or skewed features will look brilliant offline and fail online.

Stage 3 · Metrics and bias

The utility target should match the product: maximize long-term retention and healthy engagement, not raw clicks. A simple first version is utility = click/watch/return value - hide/report/complaint cost, with diversity, freshness, and safety as guardrails. Offline metrics are proxies for this utility; online retention decides whether the proxy is honest.

Offline labels come from old ranking logs, which are biased by position and selection. A clicked item was both relevant enough and exposed high enough to be seen. Missing clicks on unshown items mean nothing.

Layer	Offline metric	Online guardrail
Retrieval	Recall@K against clicked/heldout positives	ANN p99, index freshness, candidate diversity
Ranker	NDCG@K, log-loss, calibration	CTR, watch time, hides, retention, safety
Reranker	Diversity/freshness constraints satisfied	Creator concentration, filter bubbles, complaints
Exploration	Coverage and uncertainty buckets	Short-term metric loss capped, cold-start lift

Use randomization buckets or intervention harvesting so the system occasionally learns outside the old policy's favorite items. For debiasing details, see position and selection bias.

Make the log schema usable for 10a's IPS/DR estimators: log request_id, user/context, full candidate set, shown items, rank positions, scores, policy version, propensity/exploration probability, feature versions, index version, and delayed outcomes. Without propensities, offline replay can still compute NDCG on exposed items, but it cannot honestly estimate what a new policy would have done on items the old policy rarely showed.

Stage 4 · Cold start and exploration

New users and new items have sparse interaction history, so pure collaborative filtering under-serves them. The design needs a cold-start lane:

New item: content embedding from text/image/video, creator priors, freshness boost, exploration quota.
New user: onboarding interests, geo/language/device priors, session behavior, contextual bandit exploration.
System cold start: editorial/heuristic seed, popularity with diversity caps, rapid feedback collection.

Exploration is not a modeling flourish. It is the data acquisition mechanism that prevents the feedback loop from only learning about things it already showed. At this traffic scale, small percentages are huge data streams: 50k req/s · 20 slots · 1% = 10k explored impressions/s. That is enough to learn quickly, but also enough to harm the product if the exploration policy is sloppy. Put exploration behind guardrails: per-segment caps, safety filters, short-term metric loss limits, and cold-start lift tracking.

Stage 5 · HA and degradation

A ranking service must degrade rather than fail. The graceful ladder:

Failure	Degradation	User impact
Reranker slow	Skip rerank, serve ranker top-K with simple diversity	Slight quality drop
Ranker slow	Use cached/popular rank or lightweight model	Less personalized feed
Feature store partial outage	Use default/stale features with freshness tag	Quality drop, service remains up
ANN index unavailable	Fallback to popular/following/fresh candidates	Less relevant but nonempty feed

Final design

Retrieve ~1000 candidates with two-tower ANN inside 30 ms; hydrate features in bulk with freshness/missing-rate SLOs and point-in-time correctness; rank with a calibrated multi-objective model; rerank for diversity, safety, and exploration; log impressions, candidates, scores, propensities, index versions, feature versions, and positions so offline training can correct bias. Monitor p99 by stage, ANN recall, feature freshness, calibration, skew, slice metrics, and online retention.

What this case teaches

The wall is candidate recall under p99 latency. A ranker cannot recover items retrieval missed.
Feature freshness is part of serving. Online data planes are load-bearing, not backend plumbing.
Logged labels are biased by the old policy. Exploration and debiasing are required to learn beyond yesterday's ranker.
Recsys quality is multi-objective. Clicks, retention, hides, diversity, freshness, safety, and creator health all shape the final decision.