Statistical testing & A/B testing for ML

"Ship it" is a statistical claim. The four formulas every PM-facing MLE memorises, the variance-reduction trick that doubles experiment throughput, and the pitfalls that ship bad models with confidence.

The four numbers — Type I, Type II, MDE, sample size

Setup: a metric Y with mean μ_A in control, μ_B in treatment, variance σ² in both. Null hypothesis H_0: μ_B = μ_A. You run the experiment, observe a difference δ̂ = ȳ_B − ȳ_A, and reject or accept H_0.

The four are linked by one master equation. For a two-sided z-test on means with equal sample sizes:

For standard α = 0.05, β = 0.2: z_{0.975} = 1.96, z_{0.8} = 0.84, so z_α + z_β ≈ 2.80:

Inverting: to detect an effect of size MDE, you need N ≈ 16 · σ² / MDE² per arm.

Worked example — when do I ship?

You run an A/B test on click-through rate. CTR baseline is 5%, you want to detect a 0.5% lift (0.5pp absolute = 10% relative). What sample do you need?

For a Bernoulli with p ≈ 0.05, σ² = p(1−p) ≈ 0.0475. MDE = 0.005:

So ~30k per arm, ~60k total, plus a multiplier for ratio metrics, multiple variants, etc. If your daily traffic on a variant is 20k, that's ~1.5 days. Acceptable. If it's 200, that's 150 days. Don't bother.

CUPED — variance reduction for free

CUPED (Deng et al. 2013, "Controlled-experiment Using Pre-Experiment Data"). The idea: each user has a pre-experiment metric X (e.g. their CTR before the experiment). The experiment metric Y is correlated with X. Subtract the part of Y that's explained by X:

The variance of the new metric is:

If ρ = 0.6, you've reduced variance by 64% → effective sample size is ~2.8× bigger → you can detect smaller effects, or run shorter experiments. CUPED is "free" because the pre-experiment data is already there.

The peeking problem — sequential testing

You set up an A/B test for 14 days. After 3 days, you check: p < 0.05. Can you ship?

No. Every time you check the experiment, you re-run the hypothesis test. Standard inference assumes you check once; checking every day inflates the Type I rate.

Mitigations:

1. Pre-register the experiment length. Decide N up front; only test once at the end. Boring but correct.
2. Bonferroni / α-spending. If you peek K times, use α/K per check. Conservative; throws away power.
3. Sequential probability ratio tests / mSPRT. Use a test that's valid at any stopping time. Used by Optimizely, Microsoft ExP. The right answer if you actually want to peek.
4. Always-valid p-values (e.g., e-processes; Howard et al. 2021). Modern theory; small efficiency cost vs. fixed-horizon tests but always-valid.

Multiple testing — the Bonferroni / FDR pair

You're testing 20 variants against control. Any one has α=5% false positive rate, so the chance that at least one looks significant is 1 − (1−0.05)^{20} ≈ 64%. You will ship a random variant.

Choose based on the question. "I never want a false ship" → Bonferroni / FWER. "I'm okay with some false positives in a shortlist" → BH / FDR.

Ratio metrics and the delta method

Some metrics are ratios: CTR = clicks / impressions. The denominator varies per user; you can't just take the per-user CTR average. Two approaches:

1. Cluster by user. Treat each user's ratio as the unit; compute the variance of a per-user ratio.
2. Delta method. For a ratio of sums, the variance is approximately:
   Var(ΣY/ΣX) ≈ (1/X̄²) · [Var(Y) − 2·(Ȳ/X̄)·Cov(X,Y) + (Ȳ/X̄)² · Var(X)]

The delta method is the standard derivation: linearise around X̄, Ȳ using a Taylor expansion. Pitfall: it's an approximation, breaks for highly skewed denominators (e.g. heavy-tailed impressions).

Heterogeneous treatment effects (HTE)

The overall lift is +0.3%. But:

• New users: +2% (love it).
• Power users: −1% (hate it).
• Mobile: +0.5%.
• Desktop: 0%.

HTE methods estimate τ(x) = E[Y | T=1, X=x] − E[Y | T=0, X=x] as a function of features x. Standard approaches:

• Subgroup analysis: pre-register subgroups, test each (with multiple-testing correction).
• Causal forests (Athey & Wager 2019): random forest variant that estimates τ(x).
• Meta-learners (T-, S-, X-, R-learner; Künzel et al. 2019): regress outcomes by treatment status and take the difference.

The trap is finding HTE post-hoc and claiming significance — you've just multiple-tested across subgroups.

Counterfactual evaluation — when A/B is impossible

Sometimes you can't run an A/B: cold-start launches, irreversible changes, policy decisions. Use offline evaluation on logged data:

• Inverse Propensity Scoring (IPS). Reweight each historical outcome by 1/π_old(action), then estimate the new policy's expected outcome. Unbiased; high variance.
• Doubly robust: combine IPS with a learned reward model. Lower variance, but biased if the reward model is wrong.
• Direct method: fit a reward model on logged data, evaluate the new policy with the model. Lower variance, biased.

(Detailed coverage in the search/ads track, 07_bias_debiasing.)

Interactive · sample size calculator

Common pitfalls — every one ships bad models

Offline metric ≠ online lift

You have a new ranking model with +1% offline AUC. The A/B shows no lift (or worse). Why?

• Distribution shift. Offline data is from the previous policy. The new model creates a feedback loop (popular items get clicked more, get ranked higher next time, get more clicks…).
• Counterfactual data. Offline AUC measures ranking quality on items the old policy showed. The new policy shows different items — there's no offline data on those.
• Metric mismatch. AUC is rank quality; the A/B measures clicks (or revenue, or sessions). They're related but not identical.
• Long-term effects. Offline measures one-shot; A/B at week 2 includes return-user behaviour the offline metric ignores.

This is exactly the "offline ≠ online" gap covered in detail in search_ads_recsys/08_evaluation. The interview signal is naming the gap and proposing counterfactual evaluation (IPS / doubly-robust) as the partial mitigation.

The interview probes

1. "You want to detect a 1% relative lift on a 10% baseline. How many samples?" Absolute MDE = 0.001. σ² ≈ 0.09. N = 16 · 0.09 / 0.001² = 1.44M per arm. Big. Either tolerate a larger MDE or use CUPED.
2. "What's the cost of running 5 variants instead of 2?" Each variant needs N samples → 5N total (vs 2N for A/B). Plus multiple-testing correction shrinks the effective α → larger required N per arm. Total cost ~3× a simple A/B.
3. "Your experiment shows p=0.04 lift in week 1. Ship it?" Only if (a) the experiment was pre-registered to be analysed at this size; (b) no peeking has been done; (c) the metric is the pre-registered primary; (d) no novelty / primacy effects are likely. Otherwise, run to completion.
4. "What's the difference between FWER and FDR?" FWER controls P(at least one false discovery). FDR controls E[false discoveries / discoveries]. For exploratory analysis with many hypotheses, FDR is more powerful and the right choice. For ship-or-not decisions, FWER is safer.
5. "How do you A/B test a model that affects future model training (recommender feedback loop)?" The treatment data affects the world the control sees → spillover. Mitigations: temporal split (run for a fixed period and analyse the pre-feedback window), cluster randomisation (treat whole geographies), separate model training so each variant has its own loop.

Interview prompts you should be ready for