Short-video metrics - from swipe behavior to durable value

The book repeatedly frames short-video recommendation around per-user watch time, completion, interaction, retention, negative feedback, freshness, and ecosystem health. This lesson linearizes those ideas into a metric ladder so you can reason from one impression to one session to long-term platform quality.

1 · A short-video feed is not a click list

In search, the user expresses intent with a query and often clicks a result. In a short-video feed, the item is pushed into attention. Preference is expressed through micro-behavior: first-frame survival, swipe speed, watch depth, rewatch, like, comment, share, follow, hide, report, and return the next day.

The book's main point is that short-video metrics must match the product surface. CTR is still useful, but it is not enough: autoplay means a "view" is not the same as a deliberate click, and watch time can be raised in ways that harm long-term satisfaction.

2 · Build the metric ladder from local to global

A mature metrics answer does not list every metric with equal weight. It tells the interviewer which level is the launch decision and which levels are explanatory or veto metrics.

Define VTR before using it because teams overload the acronym. In this lesson, VTR means a video-specific valid-play or play-through rate: valid plays divided by exposures, sometimes with a duration threshold. It is not the same thing as CTR × watch_time, although teams may build separate value scores that multiply click probability by expected watch depth.

3 · Per-user watch time is central but dangerous

The book often uses per-user viewing time as the top business metric. It is powerful because it connects feed quality to retention and growth. It is dangerous because it can be gamed by addictive loops, outrage content, or overly narrow personalization.

The book also frames recommender goals as user satisfaction, platform value, content ecology, and system efficiency. So even a user-value-heavy feed should still watch commercial and operational metrics: ad CTR/revenue, creator supply health, compute cost per request, feature latency, and model-serving stability. These are usually secondary or guardrail metrics, but they belong in the metric map.

Use per-user metrics rather than raw totals when evaluating algorithm quality. Raw total watch time can rise because traffic rose, not because the ranker improved. Per-user watch time, capped user-level watch time, and segment-specific watch time give a cleaner signal.

3.1 · Whales distort the per-user mean - winsorize before you test

Per-user daily watch time is heavily right-skewed: it follows roughly a power law, not a bell curve. A small group of super-heavy users ("whales") may each watch many hours a day, while the median user watches a few minutes. The arithmetic mean is not robust to this shape. A handful of whales can dominate the average and, worse, can swing the treatment-vs-control delta in an A/B test purely through where they happen to land - not because the ranker changed for the typical user. This is the short-video face of the heavy-tail problem; the general "which test statistic" machinery (Mann-Whitney, delta-method, bootstrap) lives in lesson 08.

Worked example. Take a tiny control cell of 10 users with daily watch-minutes:

[2, 3, 4, 5, 6, 7, 8, 9, 10, 600]

Nine ordinary users plus one whale at 600 minutes (10 hours). The naive mean is

mean = (2+3+4+5+6+7+8+9+10+600) / 10 = 654 / 10 = 65.4 min

That 65.4 is a fiction: 9 of 10 users sit below 11 minutes, and the one whale contributes 600 of the 654 total minutes - about 92% of the metric comes from 10% of the users. Now suppose a neutral ranking change merely re-shuffles which cell the whale lands in, pushing that single user to control and a 5-minute user to treatment. Control's mean jumps by ~59 minutes while treatment's drops, and a flat experiment reads as a huge, "significant" win - a pure artifact of one account.

The fix is winsorization (a.k.a. capping): clip each user's value at a high percentile before averaging - in production typically the 99th or 99.9th, though with only ten users in this toy set that lands at the 90th, = 10 minutes. The whale is counted as 10, not 600:

winsorized mean = (2+3+4+5+6+7+8+9+10+10) / 10 = 64 / 10 = 6.4 min

Why winsorize rather than just drop the whale or always report the median? Three reasons. (1) Variance. The metric's variance scales with the largest values; capping the tail shrinks σ, which directly tightens the confidence interval and lifts statistical power for the same N (the MDE formula in lesson 08 has σ in the numerator). (2) Keep the user. Whales are real, valuable users - deleting them biases the population; capping keeps their presence while bounding their leverage. (3) Alternatives. A log(1 + watch_time) transform compresses the tail smoothly, and quantile regression or a trimmed mean targets the typical user directly. Pick the cap from the data (a fixed percentile such as p99) and freeze it before the experiment, so the threshold itself is not a knob you tune to get the result you want.

4 · Normalize duration-sensitive metrics

Completion rate is useful but biased by video length. A 7-second video can reach 100% completion with weak satisfaction; a 90-second tutorial may be valuable at 45% completion. The book suggests watch-time decay and completion-style indicators; the production move is to normalize by duration and surface.

A simple ranking score such as CTR × watch_time^alpha can be useful, but alpha is a product decision. Increasing alpha says: "we are willing to trade some click attraction for deeper consumption." That trade must be validated online.

5 · Multi-objective metrics mirror the user journey

The book decomposes per-user time into sub-objectives such as CTR, effective play or VTR, dwell time, completion, and interaction. Treat that as a journey model:

This is why MMoE, PLE, ESMM-style thinking appears throughout the book: different objectives share information, but each target has its own label quality, sparsity, and business meaning.

6 · Composite metrics need guardrails first

The book gives normalized and weighted composite-score patterns. They are useful for dashboards and model training, but a composite score can hide harm. Use a guardrail-first design:

1. Define non-negotiable guardrails: crash rate, latency, report rate, negative feedback, retention, policy safety.
2. Normalize each metric: min-max, z-score, quantile transform, or business baseline ratio.
3. Choose weights from product priorities, not from convenience.
4. Report components alongside the composite.
5. Validate the composite with A/B tests and long-term holdouts.

CS = CTR'^w1 × WatchTime'^w2 × Retention'^w3 × Diversity'^w4 / NegativeFeedback'^w5

A geometric form punishes any component that collapses. A weighted sum is easier to explain. Neither is objectively correct; both encode product judgment.

Mini-scenario: suppose watch time rises 1.2%, but report rate rises 8% and long-tail creator exposure falls 6%. A single composite score may still look positive if watch time has a large weight. A guardrail-first decision says "hold or segment the launch," because the score is hiding user-safety and ecosystem damage.

7 · Segment metrics prevent Simpson's paradox

The book repeatedly asks candidates to split by user groups. For short video, segment reads are not optional because the global average can hide product harm.

8 · Diagnose common metric conflicts

9 · Ecosystem-health and fairness metrics for short video

The ecosystem row of the ladder is the one teams under-instrument because it does not move next-swipe watch time. But a feed is a two-sided market: if creator-side exposure collapses into a few head accounts, supply dries up and watch time falls later. Short video makes this acute because the Matthew effect is fast - a video that gets a few extra impressions in its first hour can snowball, so the ranker's small biases compound into a winner-take-all distribution within a day. The general fairness/diversity machinery (group-fairness definitions, debiasing) lives in lesson 20 and lesson 08; here are the short-video-specific instruments.

Two definitions worth separating in an interview: demographic parity asks that exposure shares match population shares (an outcome test), while counterfactual fairness asks whether a creator or user would have been treated the same had a sensitive attribute (e.g. region) been different (a causal test). Short-video systems usually monitor the parity-style metrics above as guardrails and reserve counterfactual analysis for audits - see lesson 20.

9.1 · Worked case: the rural-content collapse and the fairness-cost ratio

A real pattern: a platform over-optimized per-user watch time, and rural/agricultural ("三农") content - which has a smaller but loyal audience - lost the early-hour exposure race and its user coverage cratered. The fix was not a model rewrite but an exposure-smoothing guardrail: cap any content category's 24-hour exposure growth at ≤ 200%, so a category cannot be starved (or runaway-boosted) in a single day. Result:

Read that as a fairness-cost ratio — business metric spent per unit of health metric gained (so lower is cheaper; this matches the convention in 08 · evaluation):

fairness-cost ratio = (−1.2% watch‑time) / (+32% coverage) ≈ 0.04

i.e. you give up about 0.04 percentage points of watch-time per 1% of coverage gained — a cheap trade most orgs take. A ratio of, say, 0.6 (lose 0.6% engagement per 1% fairness) would be a leadership call, not an automatic one.

A 27-to-1 trade is an easy yes - the watch-time loss is a rounding error against a large ecosystem recovery, and the recovered supply usually pays the 1.2% back over weeks. This is the Pareto framing the book pushes: do not treat fairness as a binary constraint, quantify what each point of it costs and decide deliberately. In China the Internet algorithm-recommendation regulation also makes a fairness floor a compliance guardrail, not an optional nicety.

10 · Cold-start metric sensitivity for short video

For a brand-new user or creator, CTR is a noisy, low-confidence signal - the sample is tiny and high-variance, so a CTR-driven launch read can swing on a handful of impressions. Short video gives you faster, denser early signals to lean on instead. The layered cold-start evaluation ladder (content-match → behavior-quality → exploration/Gini → cohort retention-gap, with a 14-30 day window) is built out in lesson 38; here are the short-video-flavored metrics that matter on this surface.

Worked sensitivity. Suppose 200 new users each see 5 impressions of a candidate creator (1000 impressions). With a 4% baseline CTR the standard error on that rate is roughly √(0.04 · 0.96 / 1000) ≈ 0.6%, so a one-point CTR "lift" is only ~1.6 SE — borderline, and easily lost once you remember it rests on ~40 clicks. Switch the read to first-3s survival: at perhaps a 55% rate the SE is actually larger, √(0.55 · 0.45 / 1000) ≈ 1.6% — but the true movements are larger too, because early-watch behavior shifts far more than click does for fresh content, so a 3-4 point change is ~2-3 SE and clears the bar. The win is not a smaller error bar — survival's is bigger — it is a better effect-to-noise ratio from a denser signal. (These SEs treat impressions as independent; with 5 correlated impressions per user the honest test is on the 200 per-user rates, which inflates the SE further and only sharpens the preference for the denser signal — see the per-user-unit warning in 33.) This is why short-video cold-start eval reaches for survival and VTR before CTR.

Interview prompts you should be ready for