Task metrics, calibration, and logged-policy bias

Lesson 10 built the LLM eval flywheel. General ML needs a sharper metric map: classification asks for thresholds and calibration, ranking asks for order-sensitive metrics, retrieval asks for recall under latency, and logged-policy systems ask whether the data is biased by the old model. Pick the wrong metric and the system optimizes the wrong product.

The eval first principle

A metric is a compressed model of the decision's cost. If the decision is thresholded, evaluate the threshold. If the decision ranks, evaluate the ranked list. If probabilities feed money or risk, evaluate calibration. If labels come from an old policy, evaluate the bias in the logging process before trusting the score.

Plain English dictionary

A metric is the scoreboard. A threshold is the score cutoff where the product takes action. Calibration asks whether scores mean what they claim to mean. Logged-policy bias means yesterday's system chose what data you got to observe, so today's training data is not neutral.

0 · Pick the metric by replaying the decision

Do not ask "what metric is popular?" Ask what mistake the product is trying to avoid.

If the system blocks or approves, measure precision/recall at the actual threshold and compare false-positive vs false-negative cost.
If the system orders a list, measure whether the best items appear near the top, then verify online because position bias distorts clicks.
If the system uses a probability as money or risk, measure calibration; a wrong probability can be worse than a slightly worse ranking.
If the system forecasts a future quantity, backtest on time splits, not random splits, because the future must stay unseen.
If the system learns from logs, ask what the old system chose not to show; missing exposure means missing labels.

1 · Match the metric to the decision

Metric cheat sheet

Precision: of the things we acted on, how many were right? Recall: of the true cases, how many did we catch? PR-AUC: precision/recall over many thresholds, useful for rare positives. Brier/log-loss: probability-error scores. ECE means expected calibration error: do predicted probabilities match observed rates? NDCG, MAP, and MRR are ranking metrics: did good items appear near the top? Recall@K: did retrieval include the right item in the top K? MAE, RMSE, and MAPE are forecast-error measures; RMSE punishes large misses more. The names matter less than the question each one answers.

Decision type	Primary offline metric	Must pair with	Common trap
Binary action	Precision/recall at threshold, PR-AUC	Cost matrix, review capacity	AUC high but deployed threshold bad
Probability used downstream	Log-loss, Brier, calibration error	Calibration plots by segment	Good ranking, wrong probabilities
Ranked list	Top-of-list ranking metrics	Position bias correction, online A/B	Optimizes clicks while hurting retention
Retrieval	Recall@K, nearest-neighbor lookup latency	Index RAM, p99 lookup, downstream ranking quality	Great ranker starved by missing candidates
Forecast	MAE/RMSE/MAPE/quantile loss	Backtest by horizon and segment	Random split leaks future
LLM answer	Task success, judge, human/A/B	Safety, cost, latency guardrails	Judge bias or benchmark contamination

Metrics are not interchangeable. ROC-AUC asks "does a random positive score above a random negative?" That is useful for ranking quality, but it says little about precision at the threshold that sends transactions to manual review. NDCG rewards correct order near the top; it says little about calibrated click probability for an auction.

2 · Thresholds are capacity decisions

For classifiers, the model outputs a score, but the system takes an action. The threshold should be set from costs and capacity, not from 0.5 by habit.

act when expected_benefit(score) > expected_cost(action)

Worked threshold example

Suppose a fraudulent transaction costs $100 if approved, and challenging a legitimate user costs $2 in friction/support. If a model score is a calibrated fraud probability, challenge when 100 · P(fraud) > 2 · P(legit). That crosses near P(fraud) = 2/(100+2) ≈ 2%. But if reviewers can only handle 1,000 cases/hour, capacity may force a higher threshold. The threshold is therefore both economics and operations.

System	Capacity constraint	Threshold consequence
Fraud review	Human reviewers per hour	Low threshold floods queue; high threshold misses fraud
Moderation	False takedown tolerance	Strict threshold protects users but risks creator harm
Medical triage	Follow-up slots and safety	Recall may dominate precision, but capacity still binds
Sales lead scoring	Sales team capacity	Top-K precision matters more than global accuracy

Operating-point failure

"AUC improved" is not enough to ship. Ask: at the deployed threshold, did precision, recall, cost, and segment performance improve? If not, you optimized an offline curve while the product uses one point on that curve.

3 · Calibration matters when scores are prices or risks

Calibration asks: among examples scored 0.7, does the event happen about 70% of the time? It matters whenever the score feeds expected value: ads, fraud, insurance, ranking auctions, medical risk, and downstream business rules.

ECE = Σ_k (n_k / N) · | empirical_rate_k − mean_prediction_k |

How to read it: bucket predictions by score range, compare the average prediction to the actual event rate in each bucket, then weight by bucket size. Example bins (n, pred, empirical): (6000,.05,.04), (3000,.20,.13), (1000,.60,.42). Then ECE = .6·.01 + .3·.07 + .1·.18 = .045, a 4.5 percentage-point calibration error. That means a threshold or bid using the raw score is mispriced even if AUC looks good.

Symptom	Why it matters	Fix
pCTR overestimated 2x	Advertiser overbids; auction economics break	Platt/isotonic/temperature scaling, segment calibration
Fraud score underconfident	High-risk transactions pass threshold	Calibrate on mature labels and recent traffic
Model calibrated globally, not by segment	One geography/device/user group harmed	Reliability plots by slice
Calibration drifts after deploy	Threshold no longer means same risk	Online calibration monitor and threshold review

Deep dive pointers: losses and calibration and evaluation under imbalance.

4 · Logged-policy bias is the ranking tax

Ranking and recommendation labels are usually logs from an old policy. The log says what users clicked after the old system chose what to show. It does not reveal what they would have clicked on items the old system buried.

Selection bias

If the old ranker rarely showed new creators, the training set says new creators rarely get clicked. A naive model learns the old ranker's exposure pattern and calls it user preference. This is how feedback loops become self-confirming.

The first correction is to log propensities: the probability the old policy used when it chose action a_i in context x_i. Then an offline estimate for a new policy reweights examples the new policy would have been more or less likely to show:

w_i = π_new(a_i|x_i) / π_log(a_i|x_i)
V_IPS = (1/n) Σ clip(w_i,c) · r_i
V_SNIPS = Σ w_i r_i / Σ w_i
V_DR = (1/n) Σ [ V_hat_new(x_i) + w_i · (r_i − r_hat(x_i,a_i)) ]

Mini-log: weights .8, .6, .5, 2.0 and rewards 1,0,1,0 give IPS=(.8+.5)/4=.325 and SNIPS=1.3/3.9≈.333. A doubly robust estimate adds a reward model baseline and only reweights the residual; with reasonable reward-model corrections this toy can land around .44. The teaching point is the trade: IPS is unbiased only with correct propensities but high variance; clipping and SNIPS reduce variance with bias; DR uses a reward model to reduce variance and remains useful when either the propensity model or reward model is decent.

Bias	Cause	Correction
Position bias	Top items get more attention	Click models, randomized swaps, position features
Selection bias	Only shown items can be labeled	Exploration buckets, IPS, doubly robust estimators
Popularity bias	Popular items get more exposure and labels	Diversity constraints, debiased sampling, slice eval
Policy feedback	Model changes the data it later trains on	Holdout traffic, replay logs, online guardrails

For mechanisms, see bias and debiasing, recsys evaluation, and causal/uplift modeling.

5 · Segment-sliced eval catches aggregate lies

A model can improve overall and harm a critical slice. Slice by user cohort, geography, device, language, content type, traffic source, new vs returning users, high-value accounts, and safety-sensitive categories.

Aggregate win hides...	Slice to check	Why
Latency regression	Long-context or low-end device traffic	p50 global hides p99 slice pain
Ranking regression	Cold-start users/items	Old users dominate aggregate volume
Calibration drift	Country, merchant, advertiser, language	Thresholds and bids misprice subgroups
Safety regression	Minor languages or rare categories	Small slices may carry high risk

Release rule

Ship only when the primary metric clears the bar, guardrails hold, and no protected or business-critical slice regresses beyond a predeclared threshold. Slice metrics are not "nice to have"; they are the only way to see Simpson's-paradox failures.

What carries forward

Metric follows decision. Thresholded action, ranked list, calibrated probability, retrieval, forecast, and LLM answer need different metrics.
Thresholds are product and capacity decisions. Evaluate the operating point, not only the curve.
Calibration is mandatory when scores feed money or risk. Good ordering is not enough.
Logged-policy data is biased by construction. Exploration, IPS/doubly robust eval, and online tests are how ranking systems stay honest.
Segment-sliced eval prevents aggregate self-deception. The model ships to slices, not to the mean.