A/B testing deep dive - from algorithm change to launch decision

The book treats A/B testing as the bridge between offline model quality and real business impact. This lesson linearizes that material: define the decision, build the metric hierarchy, randomize traffic correctly, verify the experiment, analyze segments, diagnose offline-online mismatch, and only then decide whether to ship.

1 · The experiment is a causal instrument

An A/B test is not a dashboard comparison. It is a controlled intervention: one population is served the old policy, another comparable population is served the new policy, and the metric difference is attributed to the policy change after randomization removes most confounders.

The book's recurring answer shape is simple: name the core business metric, split users by a stable hash, keep the control and treatment isolated, run long enough for significance, and analyze the result by user group and content group. The important upgrade is to treat that as a causal design, not as a statistical ritual.

2 · Start with the launch decision

Before choosing metrics, write the launch rule in a sentence. This prevents metric shopping after the test starts.

The book often names per-capita viewing time as the first-level metric for short-video ranking, with CTR, completion, and interaction as supporting metrics. Keep that hierarchy explicit: CTR is often a process signal; retention and durable watch behavior are closer to user value.

3 · Randomize at the right unit

The default split for a personalized recommender is user-ID hash bucketing. The reason is educationally important: one user sees many impressions, so impressions from the same user are correlated. If users are randomized but impressions are analyzed as independent rows, variance is understated and false positives become easy.

Use stratification when traffic is heterogeneous. The book explicitly calls out user groups such as new and old users; in practice add region, device tier, activity level, and content category. Stratification is not a license to hunt for a lucky slice. It is a way to balance important populations and interpret heterogeneous effects.

When you read many objectives and many slices, control the false-positive budget. The book mentions multi-test correction in multi-objective experiments; practically, pre-register the primary metric, mark slices as diagnostics unless powered, and use Bonferroni/FDR-style correction when you intend to make claims across many simultaneous tests.

3.1 · Switchback experiments when interference is market-wide

User-ID hashing assumes one user's treatment does not change another user's outcome. That assumption breaks when treatment and control share a finite resource. The clearest case is ads: pacing and budget loops (lesson 10) spend against a daily cap. If treatment users get a more aggressive bid policy, they exhaust the same advertiser budgets faster, so the control arm sees cheaper inventory it would never have seen at full traffic. The auction (lesson 9) is a single clearing market: both arms compete for the same slots. Marketplace supply/demand and two-sided liquidity have the same shape. Here cluster randomization does not help — the unit of interference is the whole market, and you cannot cut the market into independent clusters.

The fix is to randomize time instead of users. A switchback (time-split) experiment flips the entire system between policy A and policy B on a fixed time grid. In any window everyone sees one policy, so within-window there is no cross-arm leakage; the contrast comes from comparing A-windows against B-windows. The window — not the user — is the i.i.d. analysis unit.

The carryover hazard. Time windows are not independent the way users are: a policy active in window k changes state that bleeds into window k+1 — budgets already spent, pacing controllers mid-correction, a user mid-session. This is temporal autocorrelation, and it biases the naive contrast. The standard guard is a washout window: discard the first slice of each window after a switch and analyze only the settled tail.

4 · Prove the experiment system works before reading lift

Run an A/A test or logging-only test before the real A/B when the experiment platform, metrics, or new instrumentation changed. The goal is to prove that two identical policies produce statistically indistinguishable metrics and that sample allocation matches the planned split.

1. Assignment check: treatment and control counts match the target split; no sample-ratio mismatch.
2. Profile balance: user activity, region, device, and new-user share are similar across buckets.
3. Instrumentation check: exposure, play, click, watch-time, and negative-feedback events are logged identically.
4. Metric sanity: historical baselines line up; no sudden schema or unit changes.
5. Isolation check: users do not switch buckets across devices or sessions.

4.1 · Balance diagnostics with a numeric threshold

"Profile balance" needs a pass/fail number, not eyeballing. For each pre-experiment covariate (prior watch time, activity tier, region, device, new-user share), compute the standardized mean difference between arms:

SMD = ( x̄_T − x̄_C ) / √( ( s_T² + s_C² ) / 2 )

SMD divides the raw gap by the pooled spread, so it is unitless and comparable across covariates. The industry pass criterion is SMD < 0.1 on every covariate. Unlike a t-test p-value, SMD does not shrink as N grows — at billion-user scale a trivially small mean gap is "significant" by p-value yet harmless by SMD, so SMD is the right balance gauge.

The salt point is subtle and senior-level: if the bucketing hash uses a fixed seed, the same users land in treatment for every experiment forever, so their idiosyncrasies accumulate as a standing bias and any one test inherits the residue of the last. Rotating the salt per experiment re-shuffles the population so each test gets a fresh independent draw.

5 · Power the test around an effect worth shipping

The book mentions minimum sample size and significance tests. The production interpretation is: decide the smallest effect that is worth changing the product, then compute whether the test can detect it.

If the smallest meaningful lift is 0.5% watch time, do not celebrate a 0.03% statistically significant CTR lift that does not move watch time or retention. Statistics answers "is this likely noise?" Product judgment answers "is this worth shipping?"

5.1 · Significant but trivial: effect size, ROI, and the p=0.06 protocol

At hundreds of millions of users, N is so large that the standard error σ/√N collapses, so p<0.05 is nearly guaranteed for almost any non-zero effect. Significance becomes a noise filter that everything passes; it stops being a value test. Report two things alongside p.

1. Effect size (Cohen's d) — the lift in pooled-standard-deviation units, free of N:

d = ( x̄_T − x̄_C ) / s_pooled

2. ROI of the lift — pair the gain against the serving cost it demands (extra GPU/latency from a heavier model):

3. The p=0.06 protocol. A 5% lift with p=0.06 is not a rejection — p=0.06 still says the data favor the treatment, it just clears α=0.05 by a hair. Binary thresholding here throws away a likely-good change (a Type II error). Instead:

The engineering line: a p-value is a continuous strength-of-evidence signal, not a yes/no gate. Treat 0.06 as "promising, under-powered" and buy more evidence; do not let an arbitrary 0.05 cliff discard a 5% lift.

6 · Analyze multi-objective tests as a metric graph

A recommender rarely optimizes one target. The book's multi-objective A/B section suggests first-level metrics such as per-user watch time and second-level metrics such as CTR and completion. Think of these as a graph of mechanisms:

Read the graph in a fixed order. Primary metric first. Guardrails second. Mechanism metrics third. Segments fourth. A treatment where CTR rises but per-user watch time falls is not a win; it is a clue that the model found shallow attraction rather than durable satisfaction.

7 · Diagnose offline-online mismatch

The book repeatedly asks why offline metrics improve but online A/B does not. This is one of the highest-yield interview topics. The linear diagnosis is:

A strong answer does not say "offline is unreliable." It says exactly which assumption broke: static data, wrong target, stale features, biased logs, short duration, or system-level interaction.

When a conventional A/B test is too blunt or impossible, use the book's alternative evaluation tools. Interleaving can be more sensitive when two rankers are both strong and you need preference signal from the same user context. DID/quasi-experiments can help when a full randomized launch is infeasible, such as a regional policy or UI change, but they require stronger assumptions and careful pre-trend checks.

7.1 · The causal toolbox when randomization is impossible

Sometimes you cannot randomize at all: the change shipped to 100% of users, it is a region-level policy, or assignment is decided by a business rule, not a coin. The evaluation lesson (lesson 8) covers the estimation/analysis side of these tools (IPS, doubly-robust, the variance math); this lesson owns the design choice — which method the assignment mechanism forces on you. Pick by how treatment was assigned:

7.2 · Neyman allocation: send traffic to variance, not headcount

Stratification (Section 3) balances populations; it does not say how to split traffic across strata. The default is proportional — allocate each stratum traffic equal to its share of users. That is not power-optimal. The standard error of the overall mean is driven by the noisy strata, so the variance-minimizing split — Neyman allocation — over-samples high-variance strata:

n_h / n  ∝  W_h · σ_h

where W_h is stratum h's population share and σ_h its outcome standard deviation. Proportional allocation is the special case where every σ_h is equal. Low-activity users have the widest watch-time variance, so they deserve more traffic than their head-count implies.

Cluster-robust standard errors. One more variance subtlety tied to the design unit. You randomize by user-ID hash, but you often analyze per-impression rows — and a user's impressions are correlated. Treating those rows as independent understates the SE and manufactures false significance (the same trap Section 3 named). The fix is cluster-robust SEs, clustering at the randomization unit (the user, or the time window in a switchback). The variance inflation is roughly 1 + (m̄ − 1)·ρ, where m̄ is impressions per user and ρ the within-user correlation: with m̄ = 50 impressions and ρ = 0.2, the true variance is 1 + 49·0.2 ≈ 10.8× the naive value — a p-value off by an order of magnitude if you ignore clustering.

8 · Decide with long-term effects in mind

Short-video systems can win day-one watch time and lose week-two trust. The book explicitly warns about short-term versus long-term effects: over-exploitation can raise per-user time while damaging retention or satisfaction. Use a ladder:

Note the warmup (burn-in) window on the A/B rung: the first 24h–72h carry model-warmup and novelty transients that are not the steady-state effect, so they are dropped before the metric read — the same discipline as a switchback's washout, applied to the run as a whole.

For high-risk ranking changes, keep a small long-term holdout. The goal is not to block innovation forever; it is to learn whether the feed is creating durable value or just harvesting attention.

Interview prompts you should be ready for