Losses & calibration — why the loss decides whether the auction works

The architecture is settled (lesson 4). Now: what objective do you minimize, and does its output mean P(click) or just a ranking score? The answer is the difference between an ads stack that bids correctly and one that's wrong by 2×.

Pointwise BCE — the workhorse

The default loss for CTR/CVR prediction is binary cross-entropy on per-impression labels. For logit z = f(x) and label y ∈ {0,1}:

Why this and not something else? Because BCE is a proper scoring rule. The optimum of E_(x,y)[L_BCE] is achieved iff σ(f(x)) = P(y = 1 | x). The model is being asked to estimate the true posterior — calibration is the property BCE is solving for, not a separate goal you bolt on. Two practical consequences: (1) the output is directly usable as a probability — an auction can multiply it by a bid and get expected revenue with the right units; (2) shifting all logits by a constant changes the loss (σ is not shift-invariant), so the model is forced to learn the absolute level, not just the order.

Pairwise losses — the order is all that matters

If you only ever rank items relative to each other within a request, you don't need probabilities. You can train on preference pairs: an observed positive x⁺ (clicked) and a negative x⁻ (skipped or sampled). With scores s⁺ = f(x⁺), s⁻ = f(x⁻):

This is BPR (Rendle et al. 2009). RankNet (Burges et al. 2005, web search) is the same loss; BPR rediscovered it from the recsys community four years later.

The structural fact about pairwise losses: the loss depends only on the score difference. Add a million to every score and nothing changes. The output is a ranking signal, not a probability — it tells you "x⁺ beats x⁻" but emphatically does not tell you "x⁺ has CTR = 0.04". Post-hoc calibration is required if a downstream consumer wants probabilities.

Listwise losses — when the metric is a property of the whole list

Pairwise loss treats all pairs as equally important. But a swap at positions (1, 2) usually moves the ranking metric (NDCG, MRR) far more than a swap at (97, 98). LambdaRank (Burges, Ragno, Le, NIPS 2006; the 2010 paper is the unified survey) — and its GBDT incarnation LambdaMART, the historical state of the art for web search — fixes this by weighting each pair's gradient by the change in NDCG the swap would produce:

LightGBM's lambdarank objective is essentially this. The other listwise family treats the list as a softmax: L_sm = −log [ exp(s⁺) / Σ_j exp(s_j) ]. This is the classic matching/retrieval loss — every positive competes against the rest of the in-batch (or sampled) negatives; lesson 6 picks up the sampled-negatives side. ListNet/ListMLE go further by defining a probability distribution over permutations and minimising KL — theoretically clean, rarely used in production because of cost.

Pointwise vs pairwise vs listwise — the decision rule

Production reality at large platforms: pointwise BCE multi-task is the default. You get calibrated probabilities for every head (CTR, P(watch), P(convert | click)); a policy combines them. Listwise re-rankers may sit on top for inter-item effects — but the per-item scores feeding into them are still BCE-trained.

One production nuance worth naming: the pairwise and listwise families are not mutually exclusive in a real funnel. A common pattern (see lesson 04) is coarse-rank pairwise → fine-rank listwise: pairwise is cheap and prunes the candidate set fast (it only needs a score difference), then a listwise loss does the rank-aware fine-tuning on the survivors where the metric (NDCG) actually pays. BPR's two costs are why it stays out of fine-rank: it is sensitive to label noise (one mislabeled pair flips a gradient) and its compute grows with the number of pairs, not items.

The regression head is not "just MSE" — loss shapes per target

So far every loss above was for a binary label. But a modern multi-task ranker has heads whose target is not a coin flip: watch-time in seconds, play-depth, session duration. The lazy answer — "regression, so MSE" — is wrong often enough that the choice of loss shape per head is a senior-interview staple. The reason is always the same: the target distribution is not Gaussian, and MSE silently assumes it is (minimizing squared error is maximum likelihood under a fixed-variance Gaussian). When the truth is long-tailed, heavy-outlier, or really an ordering, MSE quietly optimizes the wrong thing. Here is the menu and when each earns its place.

Huber / smooth-MAE — outlier-robust watch-time

Watch-time per impression is dominated by a few enormous values: most sessions are seconds, but a binge-watcher leaves a video playing for 3 hours. MSE's gradient is the raw residual r = ŷ − y, so a single 3-hour session produces a gradient thousands of times larger than a normal one and drags the whole batch toward fitting that one outlier. Huber loss (a.k.a. smooth-MAE) fixes this by being quadratic near zero and linear past a transition point δ:

The point is the gradient: for |r| ≤ δ it is r (behaves like MSE — sensitive, fast convergence on the bulk); for |r| > δ it saturates at ±δ (behaves like MAE — a constant pull, so no single example can dominate).

Gamma-distribution loss — long-tailed duration done right

Huber stops outliers from dominating, but it still implicitly models the target as roughly symmetric around the mean. Watch-time and dwell-time are neither symmetric nor outlier-y so much as genuinely long-tailed — strictly positive, right-skewed, variance that grows with the mean. The principled move is to stop pretending the target is Gaussian and fit the distribution that actually generates positive skewed data: the Gamma. The head predicts the Gamma mean μ = f(x) (and optionally a shape k), and the loss is the negative Gamma log-likelihood. For fixed shape k this reduces to a clean per-example form:

The mechanism that matters: the error is effectively measured in relative terms (the y/μ ratio), not absolute seconds. Mispredicting a 10s video by 5s and a 10 000s video by 5000s incur comparable loss — both are 50% off — whereas MSE would consider the second error a million times worse and ignore short videos entirely. This is exactly the "variance scales with the mean" property you want for duration. (Poisson log-loss, L = λ − y log λ, is the discrete-count sibling — used when the target is a non-negative count like number of interactions; the source uses it for a predicted like-count head.)

Ordinal play-depth buckets — turn regression into ordered classification

Often you do not actually need the watch-time in seconds — you need to know how far the user got. Then the cleanest reformulation is to discretize the continuous target into ordered buckets at the natural milestones (25% / 50% / 75% / 100% completion) and predict, with K−1 binary heads, the cumulative question "did the user pass threshold t?":

Focal loss — sparse like / share / comment heads

Engagement heads other than click are brutally imbalanced: a share might fire on 0.1% of impressions. Plain BCE drowns in the easy negatives — the 99.9% of impressions the model already confidently calls "no share" each still contribute a small loss, and in aggregate they swamp the gradient from the rare positives. Focal loss (Lin et al. 2017) multiplies BCE by a modulating factor that fades out examples the model already gets right:

YouTube watch-time-weighted logistic regression — the trick that makes a classifier predict a duration

There is an elegant alternative to a regression head entirely, from YouTube's ranking paper: keep a logistic-regression (BCE) classifier, but weight each positive example by its watch-time. Negatives (no click) get weight 1; a positive that was watched for T_i seconds gets weight T_i. The remarkable result is that the learned odds of this weighted classifier approximate E[\text{watch-time}] — so you serve e^{z} instead of σ(z) and read off expected watch-time directly, while keeping all the machinery (and calibration behavior) of a BCE classifier.

Why it works: the weighted logistic regression learns odds equal to the sum of positive weights over the count of negatives. With N impressions, k of them clicked-and-watched:

A bridge to multi-objective fusion (lesson 12): uncertainty weighting

Every head above produces its own loss L_i. The ranker trains on a single sum — but the heads are on wildly different scales (a Gamma watch-time loss in the tens, a BCE share loss near 0.01) and have wildly different noise (instantaneous swipe-speed is noisy; 7-day completion is clean). A fixed weighted sum Σ w_i L_i forces you to hand-tune the w_i, and the noisy head will dominate the gradient just because its loss is numerically larger. Uncertainty weighting (Kendall et al. 2018) makes the weights learnable by treating each task's noise σ_i as a parameter:

The effective weight on head i is 1/2σ_i²: a head the model finds hard (large σ_i) is automatically down-weighted. The log σ_i term is the regularizer that stops the cheat of sending every σ_i → ∞ to zero out all losses — growing σ costs log σ.

This is only the doorway. The heavier machinery for trading off competing heads — MMoE / PLE gating, GradNorm (balancing by gradient magnitude), PCGrad / CAGrad (projecting away conflicting gradients), the seesaw effect, and online dynamic weighting — lives in lesson 12. The per-head loss shapes here are the inputs to that fusion; do not conflate the two.

What "calibrated" actually means

A binary classifier p̂(x) is calibrated if, for every predicted probability level p:

Operationally: take all impressions where the model predicted between 0.29 and 0.31; compute the empirical click rate; if it's near 0.30, the model is calibrated in that bucket. Repeat for every bucket.

• Reliability diagram — bucket predictions into K bins (10–20); plot empirical click rate vs mean predicted probability. Perfect calibration is the diagonal.
• Expected Calibration Error — ECE = Σ_k (n_k / N) · |emp_k − pred_k|. Single-number summary.
• Calibration ≠ accuracy. AUC is invariant to any monotone transform of the score; calibration cares about the absolute level. Independent dimensions.

Reliability diagrams and ECE are the offline calibration metrics; how they sit next to AUC/GAUC, log-loss, and the online A/B counterparts (and why a head can improve offline yet move no online metric) is the subject of lesson 08.

Why calibration matters for ads (and less for recsys)

The ads auction's expected revenue per impression is, roughly,

If P(click) is off by a factor of 2 systematically, eCPM is off by a factor of 2. The auction picks the wrong winner; the second-price clearing rule charges the wrong CPC; pacing controllers (which use absolute CTR to bid-shade against a budget) misfire. Calibration is the difference between charging the right CPC and charging double or half — production ads stacks routinely have a team owning the calibration layer alone.

For a pure recsys feed, the only thing that matters downstream is the order. A miscalibrated model that ranks correctly still produces the correct feed, so recsys teams sometimes get away with pairwise/listwise losses where ads teams cannot. Said another way: ads stacks consume probabilities; recsys feeds consume permutations.

Why models miscalibrate in the first place

• Sampled negatives without log-Q correction. If training replaces "all negatives" with a sample, the logits are biased by the log of the sampling probability (lesson 6).
• Class imbalance + downsampled negatives. Production click rates are 1–3%, but negatives are routinely downsampled at training time. If the prior correction is omitted at serve, every prediction is offset.
• Feature drift between train and serve. When a feature is missing/stale at serve, the model falls back to its training-prior estimate.
• Adversarial logging (selection bias). Logged training data over-represents items the previous policy already ranked at the top. IPS-uncorrected training inherits this bias (lesson 7).
• Distillation and pseudo-labels. A teacher's sharp predictions become hard labels for the student; the student inherits over-confidence.

Post-hoc calibration techniques

A useful axis: what distortion can each method fix? Temperature only stretches the logit axis — it fixes overconfidence but cannot translate the curve to fix a prior shift. Platt has a bias term, so it can. Isotonic can fix non-monotone bends that Platt's two parameters can't bend around. The widget below makes this concrete.

Interactive · the calibration explorer

Pick a failure mode and a calibrator. Watch the reliability diagram and ECE update. The point: different distortions need different fixes.

Interview prompts you should be ready for