Evaluation — offline metrics and A/B tests

You've built the ranker. Now the harder question: is it actually better? Offline metrics filter what gets tested. A/B tests decide what ships. Senior candidates quote both numbers when they discuss a launch.

Offline metrics — what each actually measures

Every offline metric is a summary statistic on logged predictions. The choice of metric is a claim about what kind of mistake matters. Wrong claim, wrong prediction of the A/B test.

Metric	Definition	Measures	Blind to
ROC-AUC	P(score(x⁺) > score(x⁻)) over random pos/neg pair.	Pure rank quality. Threshold-free.	Calibration. Magnitudes. Class imbalance.
PR-AUC	Area under precision-recall curve.	Rank quality when positives are rare. More sensitive to imbalance than ROC-AUC.	Calibration. Comparison across datasets with different base rates.
Log-loss	−[y log p̂ + (1−y) log(1 − p̂)]	The training loss. Sensitive to calibration — confidently-wrong predictions are heavily penalised.	Rank-only improvements when calibration is already tight.
NDCG@K	DCG/IDCG, with DCG = Σ (2^rel_i − 1)/log₂(i+1).	Graded relevance with position discount. Standard for 5-star / ad-quality-bucket labels.	Calibration. Score gap between adjacent positions.
MAP	Mean over queries of average precision.	Binary-relevance rank quality averaged across queries.	Graded relevance.
MRR	1 / rank_of_first_relevant, averaged.	Single-correct-answer tasks (definitional search).	Everything past the first hit. Useless for browsing.
Recall@K	Fraction of relevant items in top K.	The only metric retrieval should care about.	Order within the K. Precision.
ECE	Frequency-weighted average of \|emp − pred\| over bins (each bin weighted by the number of predictions falling in it).	Calibration — whether p̂ = 0.3 matches a 30% empirical rate.	Rank order. A perfectly calibrated random predictor has ECE ≈ 0.

Common interview misstep: treating AUC and log-loss as interchangeable. AUC is invariant to monotone transforms — great AUC and broken calibration is fatal for any auction or threshold-dependent system. Log-loss penalises miscalibration directly. Downstream is an auction (lesson 9)? Care about log-loss and ECE. Pure-sort feed? AUC and NDCG.

Which metric for which stage

Retrieval: recall@K — the only metric that matters. Order doesn't, the ranker fixes that.
Ranking: NDCG@K (if graded labels) or AUC (if binary), plus log-loss and ECE if the score feeds an auction.
Re-ranking: list-level metrics — NDCG with a diversity discount, list-CTR, or a learned listwise loss. Per-item metrics miss the inter-item effects that re-rank exists to handle.

GAUC — the metric global AUC should have been

Global AUC has a subtle defect that is the leading cause of "offline AUC up, online flat." AUC is computed over all positive/negative pairs in the dataset — including pairs that span different users. So a model can score every popular item above every unpopular item across the whole corpus, win those cross-user pairs in bulk, and post a high AUC — while the ranking within any single user's candidate list (the only ordering that user ever sees) is mediocre. The user never sees a list mixing their items with a stranger's, so cross-user pairs are rank comparisons nobody experiences.

GAUC (Group AUC) fixes the estimand: compute AUC per user on that user's own impressions, then average with a per-user weight (usually impression or click count):

GAUC = ( Σ_u w_u · AUC_u ) / ( Σ_u w_u ), w_u = impressions (or clicks) of user u

Only within-user pairs enter the average, so GAUC measures exactly what the product does — order this user's candidates well. It is the industrial standard for CTR ranking; the Alibaba DIN paper popularised it precisely because global AUC was a poor predictor of online CTR while GAUC tracked it.

Worked numbers — where AUC and GAUC diverge (carry this one)

Two users, two impressions each, label = click (1) or no-click (0). Same model scores all four impressions.

User	Item	Label	Model score
A (heavy)	a1	1 (click)	0.40
A (heavy)	a2	0	0.50
B (light)	b1	1 (click)	0.80
B (light)	b2	0	0.90

Global AUC. List all positive×negative pairs and count how often the positive outscores the negative. Positives are {a1=0.40, b1=0.80}; negatives are {a2=0.50, b2=0.90}. Four pairs:

(a1 0.40 vs a2 0.50) → pos < neg ✗ within-user A (a1 0.40 vs b2 0.90) → pos < neg ✗ cross-user (b1 0.80 vs a2 0.50) → pos > neg ✓ cross-user (b1 0.80 vs b2 0.90) → pos < neg ✗ within-user B

Global AUC = 1 correct / 4 = 0.25. (The one "win" is the cross-user pair b1>a2 — an ordering no user ever sees.)

GAUC. Score each user on their own pair only. User A: positive a1=0.40 vs negative a2=0.50 → pos<neg → AUC_A = 0. User B: b1=0.80 vs b2=0.90 → pos<neg → AUC_B = 0. Weight by impressions (2 each): GAUC = (2·0 + 2·0)/(2+2) = 0.0.

The model ranks the click below the non-click for both users — within-user ranking is the worst possible (GAUC 0) — yet global AUC reads 0.25, lifted entirely by a cross-user pair. Flip the construction (give each user the right within-user order but jumble scores across users) and you get GAUC = 1.0 with global AUC well below 1.0. The two metrics can point in opposite directions; GAUC is the one that predicts the A/B test. Practical note: users with only positives or only negatives have an undefined AUC_u — drop them (or assign 0.5) and report what fraction you dropped.

Evaluating cold-start — when CTR itself is unreliable

Every metric above assumes a warm, logged population. For a new user there are almost no clicks yet, so CTR is a high-variance estimate on a handful of impressions — a difference of two clicks swings it wildly, and the model that "wins" cold-start CTR offline is often just the luckier one. You cannot evaluate cold-start with the warm toolkit; you build a layered eval, each layer answering a different question, and you extend the window to 14–30 days because new-user feedback (does this user come back?) arrives late.

Layer	Metric	What it certifies / why CTR can't
1 · Content match	Tag-coverage accuracy; cross-domain transfer AUC (pretrain on a data-rich domain, measure discrimination in the target domain on the few labels you have).	Can the model discriminate at all for this user before behaviour exists? Transfer AUC borrows signal from a populated domain; raw CTR has no labels to compute.
2 · Behaviour quality	VTR (view-through rate: plays > 3 s, to filter mis-taps), deep-conversion rate (like/save/follow).	A click on a cold item is mostly thumbnail curiosity. VTR/deep-conversion are denser, higher-signal-per-event proxies that survive sparse data.
3 · System health	Exploration-exposure ratio (share of impressions to fresh items); exposure Gini across the candidate pool.	A cold-start policy that quietly stops exploring scores fine on the items it does show. Gini catches the collapse into a popular-item rut.
4 · Cohort comparison	7-/14-/30-day retention-gap: new-user retention minus the main-pool baseline, expected to converge.	The only layer tied to the real goal. A healthy cold-start narrows the gap over the window; a stuck gap means onboarding is failing regardless of CTR.

The VTR-vs-completion trap

Layer 2 looks safe until you over-optimise it. Reward VTR (play > 3 s) alone and you breed "opening-shock" clickbait — a loud first second that clears the 3 s bar, then the user bounces. The metric rises while satisfaction falls, the same proxy-divergence failure as clickbait CTR one section up. Balance VTR with a completion-rate floor (or weight by watch fraction) so "grabbed attention for 3 s" can't masquerade as "served a good video." Cross-link: cold-start modelling (cross-domain transfer, meta-learning) lives in the cold-start lesson; here we only measure it.

The fundamental problem with offline metrics

Every offline metric is computed on logged data generated by the old ranker. Two consequences:

You can only score items the old ranker showed. If the new ranker would surface an item the old one never put in front of users, there's no label. Offline metrics are biased toward changes that re-order items the old policy already surfaced.
The metric is a proxy. CTR-optimised offline metrics can rise while watch-time falls. NDCG can rise while diversity collapses. Every offline metric is one summary of behaviour; the business cares about long-term retention.

Worked numbers — how NDCG rises while watch-time falls

This is the single most-asked evaluation question, so carry a concrete version. A new ranker promotes clickbait thumbnails. Score it two ways on the same week of traffic:

Metric (top-10 feed)	Old ranker	New ranker	Δ
Offline NDCG@10 (label = click)	0.420	0.441	+5.0%
Online CTR (A/B)	5.0%	5.6%	+12%
Online watch-time / session	14.0 min	12.3 min	−12%
7-day retention	baseline	−0.8 pp	down

The offline metric and CTR both rise because the label is click and the new ranker is genuinely better at producing clicks. But users click, watch 5 seconds, bounce, and the session ends shorter — so watch-time and retention fall. The offline metric was never wrong about clicks; it was optimising a proxy that diverges from the goal. Two fixes that an interviewer wants to hear: (1) change the offline label to watch ≥ 30s or a weighted multi-objective so the proxy tracks the goal (lesson 12); (2) gate launches on a watch-time / retention guardrail, not CTR alone.

Offline metrics are a filter, not a decision. They tell you whether the model earns A/B-test traffic. A/B tests tell you whether to ship.

Counterfactual / off-policy evaluation

If you've trained a new ranker but haven't A/B tested, you can estimate online performance from logged data via importance-weighted estimators (the IPS machinery of lesson 7). For a logged tuple (context, action, reward) with old-policy propensity π_old(a | context):

Ê[reward | π_new] = (1/N) · Σ reward · π_new(a | context) / π_old(a | context)

Requires logged propensities. You needed to write down π_old(a | context) at serving time. Post-hoc estimation adds bias.
High variance. When π_new and π_old disagree strongly, weights blow up. Doubly-robust estimators (Dudík et al. 2011) combine IPS with a learned reward model — unbiased if either propensity or reward model is correct.

Worked numbers — IPS, and why variance explodes

Suppose the old policy showed a particular ad to a context with propensity π_old = 0.10 (10% of the time), and in those logged impressions the reward (a click) was observed. The new ranker would show that ad with π_new = 0.50. The importance weight is 0.50 / 0.10 = 5: each logged click for this ad counts as 5 clicks in the estimate, because the new policy would have surfaced it 5× as often. That is the correction working as intended. Now the danger: an ad the old policy almost never showed, π_old = 0.001, that the new policy loves, π_new = 0.5 — weight = 500. A single lucky click becomes 500 estimated clicks, and the estimate is now hostage to one log row. That is the high-variance failure: when π_new puts mass where π_old put almost none, the estimator's variance blows up. Mitigations: clip weights at, say, 10 (trades a little bias for far less variance), or use the doubly-robust estimator, which leans on a learned reward model exactly where the propensity ratio is unreliable.

Off-policy evaluation never replaces an A/B test for shipping. It reduces how many candidate A/B tests you have to run.

When you can't randomise — the causal-evaluation toolbox

IPS estimates a counterfactual from logged data, but plenty of effects can't be A/B tested at all: a change rolled out by region, a feature gated behind a threshold, a launch confounded by a holiday. Then you're estimating a causal effect from observational data, and the job is to kill the confounder — the variable that drives both who got treated and the outcome — so the measured difference is the policy's effect, not a selection artefact. Each method below kills a different confounding structure; naming the method without the scenario it fits is the junior tell.

Method	Use when…	What it assumes / kills
DID (difference-in-differences)	A policy rolls out to one group at a known time; you have a comparable untreated group and pre-period data for both.	Differences out any time-invariant gap between groups and any common time trend. Precondition: parallel trends — absent treatment, the two groups would have moved in lockstep. Validate it on the pre-period: if the lines weren't parallel before treatment, DID is invalid.
PSM (propensity-score matching)	Observational treated/untreated users differ on observed covariates (heavy users self-selected into the feature).	Match each treated user to controls with the same propensity P(treated \| covariates), then compare. Kills observed confounding only — assumes no unobserved confounder (selection on observables).
RDD (regression discontinuity)	Treatment is assigned by a threshold rule (loyalty perk for spend > $100; promo for score ≥ cutoff).	Users just above vs just below the cutoff are near-identical except for treatment, so the jump at the boundary is causal. Local effect only (near the threshold); needs enough mass around it.
IV (instrumental variables)	An unobserved confounder you can't match on, but you have an instrument that shifts treatment without touching the outcome directly.	Uses only the treatment variation driven by the instrument. Strong, untestable assumptions (relevance + exclusion); fragile, but the one tool for unobserved confounding.
Double ML	Observational data with many covariates and a non-linear outcome/treatment relationship — the modern default for observational effect estimation.	Two-stage: ML models the nuisances (propensity + outcome) flexibly, then a debiased (orthogonalised, cross-fitted) score estimates the effect — robust to ML overfitting in stage one.
Uplift / CATE (causal forest, meta-learners)	You want the heterogeneous effect — which users a change helps, hurts, or leaves flat — not one average.	Estimates a per-segment treatment effect τ(x) instead of one ATE. The right tool when an average lift hides a subgroup it's quietly harming.

The parallel-trends failure, concretely

You roll a ranker change to Region A; Region B is the control. Pre-launch, A's watch-time grew +3%/quarter and B's +1%/quarter — not parallel, A was already pulling away. Post-launch A is +5%, B is +2%, so naive DID credits the launch with (5−2) − (3−1) = +1 pp. But A's pre-trend alone would have widened the gap ~+2 pp/quarter, so the "+1 pp" is plausibly the pre-existing divergence, not the ranker — the effect could even be negative. Always plot both groups' pre-period; a non-flat pre-trend means reach for synthetic control or a different design. Cross-link: full experiment design — switchback, cluster randomisation, A/A — lives in lesson 33; here we only cover the measurement side once randomisation is off the table.

A/B testing — the gold standard, with traps

Randomly assign users to treatment (new ranker) and control (old ranker). Measure the metric in each arm. Run a statistical test. The mechanics are simple; the traps are subtle and most of the questions interviewers ask are about traps.

Power and minimum detectable effect

For a two-sample test of means with per-arm sample size N and per-user metric standard deviation σ, the MDE at 80% power and 5% two-sided significance is

MDE ≈ (z_α/2 + z_β) · σ · √(2/N) ≈ 2.8 · σ · √(2/N)

(For a Bernoulli click-rate metric, σ² = p(1−p), so MDE ≈ 2.8·√(2p(1−p)/N) per arm.)

This is the trap that gets juniors. Detecting a 0.5% relative lift on a 5% baseline CTR needs millions of users per arm. Underestimating sample size by 10× is the modal mistake. The reflex when asked "how long would you run this?" is to compute the MDE first.

Type I and Type II errors

Running 20 simultaneous tests at α = 0.05 produces, in expectation, one false positive. Mitigations:

Bonferroni / Holm-Bonferroni: divide α by the number of tests; controls FWER but kills power.
FDR (Benjamini-Hochberg): controls expected false-discovery rate, not family-wise error. Right knob for screening many variants.
Always-valid p-values / sequential testing (Howard et al. 2018): peek continuously without inflating Type I. Mandatory at any lab running thousands of experiments.

Type II — under-powered tests missing real effects — is the quieter failure. Teams running 1-week tests on low-traffic surfaces routinely declare "flat" when the true effect was real but undetectable.

Which test for which metric

"Run a statistical test" hides a choice that interviewers probe directly. The metric's type dictates the test; pick wrong and your p-value is meaningless.

Binary metrics (CTR, conversion) at large N: a two-proportion z-test, equivalently a χ² test on the 2×2 table. z = (p̂_t − p̂_c) / √(p̄(1−p̄)(1/n_t + 1/n_c)) with pooled p̄. The CLT makes the z approximation tight once expected counts per cell exceed ~10.
Continuous per-user means (watch-time per user, revenue per user): Welch's t-test — never assume equal variance across arms, the treatment usually reshapes the distribution. At A/B-test N the t and z coincide; the Welch correction is about the unequal-variance, not the df.
Heavy-tailed / non-normal metrics where you can't lean on the CLT at your N (sessions, spend with a few whales dominating): Mann-Whitney U or a permutation test. These test a shift in distribution / stochastic dominance, not the mean — know that you've changed the estimand.

The ratio-metric trap — the one that trips people up

Per-user watch-time is fine: the randomization unit (user) is the metric unit, so users are i.i.d. and the variance is textbook. But ratio metrics — clicks-per-impression aggregated across all impressions, where you randomize on users but the denominator counts impressions — break the i.i.d. assumption. One heavy user contributes hundreds of correlated impressions; treating impressions as independent under-estimates the variance and manufactures false significance. Fix: compute the variance with the delta method, Var(X̄/Ȳ) ≈ (1/Ȳ²)·Var(X̄) − 2(X̄/Ȳ³)·Cov(X̄,Ȳ) + (X̄²/Ȳ⁴)·Var(Ȳ) where X̄, Ȳ are the per-user numerator and denominator means, or sidestep the algebra with the bootstrap below. This is the single most common A/B-analysis bug at scale.

Bootstrap is the general-purpose fallback when there's no clean closed-form variance: resample the randomization units (users, with replacement), recompute the metric on each resample, and read the confidence interval straight off the resampling distribution. It handles arbitrary and ratio metrics, capped/winsorized metrics, and weird estimators without any distributional assumption — at the cost of compute. When in doubt about a custom metric's variance, bootstrap it.

Metric type	Example	Test
Binary / proportion	CTR, conversion rate	Two-proportion z-test (≡ χ²)
Continuous mean, per unit	Watch-time per user	Welch's t-test (unequal variance)
Ratio (denominator ≠ randomization unit)	Clicks per impression, aggregated	Delta method for variance, or bootstrap
Heavy-tailed / non-normal	Revenue with whales, session counts	Mann-Whitney U / permutation test

Variance reduction — the cheapest power upgrade

Technique	How it works	Reduction
CUPED (Deng et al. 2013)	Replace Y by Y − θ(X − E[X]) with pre-experiment covariate X and θ = Cov(Y,X)/Var(X).	30–50% on most user-level engagement metrics. Free power.
Stratification	Pre-stratify by covariate (country, device, heavy/light user); analyse within strata.	Modest; composable with CUPED.
Triggered analysis	Only count users who hit the changed code path.	Large when trigger rate is low. Careful definition needed to avoid post-treatment selection bias.

CUPED is standard at every major experimentation platform. Naming it signals you've shipped experiments at scale.

Network effects, novelty, primacy

Effect	What goes wrong	Mitigation
Network interference	Treatment users influence control users (sharing a video). Effect leaks across the assignment boundary, biasing toward zero.	Cluster-level randomization, ego-cluster designs, side-by-side surfaces, or bound the bias.
Novelty bias	Users initially engage more with anything new. The week-1 lift is partly novelty, not the model.	Run for weeks; report the trajectory. Or burn-in: discard the first N days.
Primacy effect	Opposite — users initially worse because they're used to the old thing, then adapt. Underestimates real effect.	Same — long enough experiments, examine the trajectory.
Twyman's law	"+50% CTR" wins are almost always instrumentation: misrouting, double-counting, broken control. Real ranker wins are 0.1–2%.	SRM checks, A/A tests, instrumentation review before believing any large effect.

Two sanity checks to run before you trust any A/B result

SRM (Sample Ratio Mismatch). You assigned 50/50, but the realized split is 49.8% / 50.2% — is that fine, or a bug? Run a χ² test on the observed counts against the intended ratio. With millions of users even a 50.4/49.6 split fails the test, and a failing SRM means the assignment or logging is broken (a redirect that drops slow clients, a crash that disproportionately hits one arm). An SRM invalidates the experiment — the arms are no longer comparable, so the lift is meaningless. Check it first, before reading any metric.

Day-of-week seasonality. Traffic on Tuesday is a different population and a different mood than Saturday. If you run Mon–Fri only, weekday behaviour confounds the result and the variance is understated. Always run for a full week or a multiple of seven days so each arm sees two complete weekly cycles; otherwise a launch that looks like +1% might just be "we happened to measure over a high-engagement weekend."

Long-term metrics

Short-term CTR is easy; long-term DAU and retention are what the business cares about. The two regularly disagree — a CTR-optimised ranker that shortens sessions (clickbait → dissatisfied user closes the app) wins the short experiment and loses the company.

Standard pattern: a long-term holdout keeps a fixed cohort on the pre-launch experience for 3–6 months after the regular A/B test concludes; compare retention and revenue trajectories. Most orgs run one product-wide holdout with new launches stacking against it, rather than one per experiment.

Measuring fairness — the metrics, not the fix

An engagement-maximising ranker has no reason to spread exposure fairly: feeding the head of the distribution is locally optimal, so the long tail (new creators, minority-interest content, rural-skewed topics) silently starves. That's a Matthew effect, and in several jurisdictions it's now a compliance line, not a nice-to-have. Fairness evaluation is the measurement layer that makes the harm visible; the mitigation (re-ranking, exposure constraints, adversarial debiasing) is a separate problem covered in lesson 20. Here we only quantify it. Two families to keep straight:

Demographic parity — a purely statistical equality of outcomes across groups (equal exposure share, equal CTR). Easy to compute, but blind to whether a gap reflects genuine preference vs model bias.
Counterfactual fairness — would this user's recommendations change if only their group attribute flipped, all else equal? A causal notion (it isolates the attribute's effect), much closer to "is the model actually discriminating," and much harder to measure.

Metric	Definition	Catches
Exposure Gini	Gini coefficient of impressions across items/creators (0 = perfectly even, 1 = one item takes everything). Common health target ≤ 0.3.	Concentration / Matthew effect — a few items eating all exposure while the tail gets nothing.
Group-exposure KL	KL(exposure-share ‖ population-share) across groups: Σ_g e_g · log(e_g / p_g).	A group under-served relative to its prevalence. 0 only when exposure share matches population share for every group.
Group-CTR disparity	Unfairness coefficient (CTR_max − CTR_min) / CTR_avg across groups.	Outcome inequality — the model serves some groups well-matched content and others poorly, even at equal exposure.
Fairness-cost ratio	(% engagement lost) / (% fairness gained) when you intervene — the slope of the Pareto trade-off.	How expensive fairness is here. A low ratio means a cheap win; a high ratio is a real business decision to escalate, not an eng call.

Worked numbers — the rural-content collapse

A ranker is retuned for watch-time. Rural ("三农"-style) creators were 8% of the catalogue and historically reached 25% of users; population share of rural-interested users is ~22%. After the launch their reach drops to 17%.

Signal	Before	After	Read
Creator-exposure Gini	0.31	0.44	past the ≤0.3 line, worsening
Rural-reach vs 22% pop.	25%	17%	now under-served
Per-user watch-time	baseline	+1.2%	the "win" that caused it

The KL term for this group moves from 0.25·log(0.25/0.22) = +0.032 (slightly over-served) to 0.17·log(0.17/0.22) = −0.044 per-group contribution — it flips from a positive to a negative deviation, the quantitative signature of a group going from over- to under-served. Mitigation (an exposure-smoothing rule, e.g. 24 h exposure growth capped at +200%) belongs to lesson 20; once applied, the fairness-cost ratio tells you whether it was worth it. Reported outcome of exactly this fix: rural creator reach recovered +32% for only −1.2% per-user watch-time — cost ratio ≈ 1.2/32 ≈ 0.04, a cheap trade most orgs take. A ratio of, say, 0.6 (lose 0.6% engagement per 1% fairness) is a leadership decision, not an automatic one.

Interactive · A/B test power calculator

Plug in a baseline metric, a desired lift, and your daily traffic. The widget tells you the MDE at your current sample size, how many days you need to detect the target lift, and what a Bonferroni adjustment does to your significance threshold if you're running many tests in parallel.

Sizing an experiment

All inputs are per-arm. The MDE formula assumes a two-sided z-test on the difference of proportions with σ² ≈ p(1−p) per user. Real platforms use cluster-robust or CUPED-adjusted variance; the order of magnitude is the same.

baseline CTR (%): 5.0 target relative lift (%): 1.0 daily users per arm (thousands): 200 observed lift so far (%): 0.40 days running: 7 simultaneous tests: 1

N per arm so far

—

MDE @ 80% power (rel %)

—

days to reach target lift

—

p-value (two-sided)

—

Bonferroni threshold

—

verdict

—

Reading

—

Trade-offs at a glance

	AUC	Log-loss	NDCG@K	A/B test
Latency to result	Minutes (offline)	Minutes	Minutes	1–8 weeks
Cost	Trivial	Trivial	Trivial	Engineering + opportunity cost of traffic
Calibration-sensitive?	No	Yes	No (rank only)	Indirectly (auctions, thresholding)
Position-bias-sensitive?	Yes — logged labels are biased	Yes	Yes	No — randomisation breaks the link
Counterfactual to old policy?	No (logged data only)	No	No	Yes — by construction
Decision risk	Misleads launches	Misses rank issues	Misses calibration	The actual launch criterion

Where junior candidates trip

Asked "how would you evaluate the ranker?" a junior candidate names one offline metric and stops. A senior candidate names the offline metric, names a calibration check, names the planned A/B test with rough sample size, and names what they would do if offline improved but the A/B test was flat. The last part is the test — the interviewer wants to know whether you understand that offline metrics are advisory.

Interview prompts you should be ready for

"Offline NDCG@10 improved 3%; A/B test is flat. Diagnose." (Probes: retrieval ↔ ranker mismatch, position bias in labels, lift on items the old ranker never surfaced, novelty masking, offline metric not matching production scoring, re-rank overrides.)
"Walk me through CUPED. Why does it work?" (Probes: pre-experiment covariate fixed before treatment so estimator is unbiased; variance reduction proportional to ρ² between pre- and in-experiment metric.)
"How do you decide how long to run an A/B test?" (Probes: power calculation from baseline and σ, novelty burn-in, weekday cycles, multiple-tests adjustment.)
"Why is AUC sometimes misleading as a model-selection metric?" (Probes: invariance to monotone transforms → blind to calibration; insensitive to where in the ranking the improvement happens; matters for auctions.)
"New ranker has big A/B lift in week 1; week 4 flat. What's happening?" (Probes: novelty decay, instrumentation broke, traffic mix shifted, feature distribution drift, contaminated holdout.)
"Test 10 ranker variants — naive Bonferroni or something smarter?" (Probes: Bonferroni is fine for small numbers but conservative; FDR for screening; sequential/bandit for adaptive allocation; tournament against a champion.)
"You can't run an A/B test. How do you evaluate?" (Probes: off-policy IPS/doubly-robust, interleaving, shadow scoring, human raters, replay on held-out logs; for non-randomisable rollouts the causal toolbox — DID with parallel-trends check, PSM, RDD for threshold rules, IV for unobserved confounders, Double ML.)
"Offline AUC is up but you suspect it's misleading for ranking. What do you compute instead and why?" (Probes: global AUC counts cross-user pairs nobody sees and can move opposite to true ranking quality; GAUC = impression-weighted per-user AUC measures within-list order, the DIN-standard predictor of online CTR; should sketch a 2-user divergence.)
"How do you evaluate a model for brand-new users when CTR is too noisy?" (Probes: CTR is high-variance on sparse data; layered eval — cross-domain transfer AUC, VTR/deep-conversion, exploration ratio & exposure Gini, cohort retention-gap; 14–30 day window; the VTR-vs-completion clickbait trap.)
"How would you measure — not fix — whether the ranker is unfair to long-tail creators?" (Probes: exposure Gini, group-exposure KL vs population share, group-CTR disparity, fairness-cost ratio for the Pareto trade-off; demographic parity vs counterfactual fairness; mitigation is a separate question.)

Takeaway

Offline metrics filter. A/B tests decide. Each offline metric encodes a specific claim about what kind of mistake matters — AUC about rank, log-loss about calibration, NDCG about graded position, recall about retrieval. A/B tests have their own failure modes — underpower, novelty, network effects, multiple-testing — and the senior signal is naming the mitigations (CUPED, sequential tests, long-term holdouts) before the interviewer prompts for them. When asked about a launch, quote both numbers: the offline lift that earned the test, and the online lift that earned the ship.