Reinforcement learning for recommendation

Every ranker so far answers one question: which item gets clicked next? That objective has a hidden, expensive assumption — that the next click is what you actually want. It isn't. This lesson recasts the feed as a sequential decision problem and confronts the three things that make RL on logged data genuinely hard.

The limit this lesson removes

Supervised ranking optimizes a myopic objective: P(click | this item, this context), one impression at a time, treating impressions as independent. But a feed is not independent impressions. Showing item A changes the user's state — their boredom, their satisfaction, whether they open the app tomorrow. The thing today's recommendation maximizes and the thing the business wants can diverge, and a one-step model cannot even see the gap. RL is the framework that makes the horizon explicit.

Why "predict the next click" is not enough

Consider an autoplay short-video feed. A greedy CTR ranker learns that outrage clips and clickbait get clicked, so it serves more of them. Click-through ticks up this week. But three weeks later daily-active-users is down, because the experience got exhausting — and no impression-level label ever recorded "this session made me close the app." The damage is spread across the future, attributed to no single action, invisible to a model whose loss only sees the immediate label.

Two structural facts break the supervised framing:

Actions change the environment. Today's recommendations shape tomorrow's user — their interests, their fatigue, their trust. The data distribution you face next is a function of the policy you run now. This is the defining feature of an RL problem and the explicit absence in supervised learning.
The objective is a sum over a horizon, not a point. You want to maximize cumulative engagement / retention / lifetime value across a session and across the user's lifetime — a return, not a single reward. A clip slightly less clickable now but that keeps the user exploring can be worth far more than a clickbait that wins this slot and costs the next ten.

Proxy gaming, stated precisely

Whenever you optimize a measurable proxy (next-click) for an unmeasurable goal (long-term satisfaction), a powerful optimizer will find the gap between them and drive into it. The cure is not a cleverer model of the proxy — it is changing what you optimize from a one-step reward to a long-horizon return. That reframing is exactly the move from supervised ranking to reinforcement learning.

The feed as a Markov Decision Process

An MDP is the five-tuple (S, A, P, R, γ). The whole lesson is the act of mapping a recommender onto these five symbols and then asking what each one costs.

MDP element	In a recommender	The catch
State S	User interest embedding, recent interaction sequence (last ~10–20 items, their watch %, like/skip), plus context: time of day, device, network.	Truly partially observed — you never see the user's actual mood/intent, only behavioral traces. Strictly a POMDP; the sequence model is your state estimator.
Action A	Which item to show next, or the Top-K slate; sometimes a continuous vector of fusion weights (e.g. how to mix watch-time vs like in the score).	Discrete action space is enormous (millions of items) and changes hourly as new content arrives — the central reason vanilla Q-learning struggles here.
Transition P(s′\|s,a)	How the user's state evolves after seeing an item — interest drift, fatigue, satisfaction.	Unknown and non-stationary. Most production RL is model-free precisely to avoid having to learn this.
Reward R(s,a)	Watch-time, completion, like, share, follow — or a return visit / next-day retention.	Sparse (follows are rare), delayed (retention is observed tomorrow), and a proxy that can be gamed.
Discount γ	Weights immediate vs future reward; γ = 0.9–0.99 typical.	γ→0 recovers the myopic supervised ranker. γ→1 values the long horizon — and explodes variance.

The Markov property demands that s_t contain everything relevant about the past — so the quality of your state representation is the validity of the whole framing. This is why the behavior-sequence models you saw in ranking and 13 · User behavior sequences (RNN/Transformer encoders over the click stream) reappear here as the state encoder: they are how you compress an unbounded history into a Markovian s_t. Hold onto this — when the serving budget bites, the per-request cost of that encoder is the first thing you will have to cut.

Return, value, and the Bellman equation

The agent maximizes the expected discounted return from each state — the cumulative future reward, geometrically down-weighted:

G_t = Σ_k=0^∞ γ^k r_t+k+1 Q^π(s,a) = E_π[ G_t | S_t=s, A_t=a ]

Q^π(s,a) is the action-value: "if I show item a in state s and behave by policy π thereafter, what total discounted reward do I expect?" That single quantity is what separates RL from CTR prediction — a CTR model estimates r (this click); Q estimates the whole tail. The recursion that ties present to future is the Bellman equation:

Q^π(s,a) = E[ r + γ Q^π(s′,a′) ] Q^*(s,a) = E[ r + γ max_a′ Q^*(s′,a′) ]

The optimal Q^* says: the value of an action is its immediate reward plus the discounted value of the best thing you can do next. Estimate Q^* well and the optimal policy is just argmax_a Q^*(s,a). Everything that follows is a way to estimate Q (value methods) or to optimize π directly (policy methods) — or both (actor-critic).

Worked numbers — why a myopic ranker picks the wrong clip

Two candidate clips for a bored user, γ = 0.9. Clip A (clickbait): immediate reward r = 1.0 (a satisfying click), but it leaves the user in a "burned-out" state whose best continuation is worth Q(s′) = 2.0. Clip B (a genuinely good recommendation): immediate reward r = 0.6, but it leaves the user engaged, with a continuation worth Q(s′) = 8.0. Apply the Bellman backup Q = r + γ·Q(s′):

Q(A) = 1.0 + 0.9 × 2.0 = 2.8 Q(B) = 0.6 + 0.9 × 8.0 = 7.8

A myopic ranker (γ = 0, or a CTR model that only sees r) picks A — it has the higher immediate reward. The horizon-aware agent picks B, worth nearly 3× as much over the session. The entire value of RL here is the second term: the continuation. Notice it also shows where RL gets hard — those Q(s′) values are themselves estimates the agent must learn, and an over-estimate that compounds across the horizon is exactly the failure Double-DQN exists to fix.

Bandits — the one-step special case

Before reaching for full sequential RL, notice the easiest case: set γ=0, or assume each action does not change the state. The return collapses to the immediate reward, the Bellman recursion disappears, and the MDP becomes a multi-armed bandit — choose among arms (items) to maximize immediate reward, learning each arm's payoff from feedback. A contextual bandit adds the state: choose the best arm given context x (the user), i.e. learn E[r | x, a].

A contextual bandit is a CTR model that knows it must explore

Estimating E[r | x,a] is exactly CTR prediction. The only thing a bandit adds is a principled answer to "should I show the item I think is best, or one I am uncertain about to learn more?" — the explore/exploit decision. That single addition is why bandits are the right first tool for cold start: a brand-new item has no clicks, so its estimated CTR is meaningless; a greedy ranker never shows it and it never gets a chance, while a bandit deliberately spends a little traffic to find out.

The classic explore/exploit policies, in increasing sophistication:

ε-greedy — exploit the best-estimated arm with probability 1−ε, pick a uniformly random arm with probability ε. Trivial; wastes exploration on obviously bad arms.
UCB (upper confidence bound) — pick argmax_a[ μ̂_a + c√(ln t / n_a) ]: be optimistic in proportion to your uncertainty. Explores arms you have tried little (small n_a), not arms that are merely bad. LinUCB is the contextual version. Regret grows only as O(log t).
Thompson sampling — keep a posterior over each arm's reward, sample from it, act greedily on the sample. Explores in proportion to the probability an arm is optimal — usually the best-behaved and cheapest to run at scale.

Interactive · bandit regret simulator

Ten items, each with a hidden true CTR. A pure-greedy policy locks onto whatever looked best early — often the wrong arm. ε-greedy, UCB, and Thompson sampling spend a little traffic learning, and overtake it. Watch cumulative regret (reward lost vs always playing the true-best arm): lower is better.

Full RL — value, policy, and actor-critic

When actions genuinely change state (the user gets bored, or hooked), you need the full horizon. Three families, each answering a different failure of the one before it.

Value-based: DQN

Q-learning learns Q^* by bootstrapping the Bellman target. DQN replaces the intractable Q-table with a neural net Q_θ(s,a) and adds two stabilizers that matter in production: an experience replay buffer (sample past transitions randomly to break temporal correlation and reuse data) and a target network (a frozen copy Q_θ⁻ for the bootstrap target, refreshed slowly) to stop the network chasing its own moving estimate. The loss is the squared Bellman error:

L(θ) = E[ ( r + γ max_a′ Q_θ⁻(s′,a′) − Q_θ(s,a) )² ]

Strength: the explicit max bootstrap makes DQN sample-efficient and naturally long-horizon; replay reuses scarce reward signal — valuable when likes/follows are sparse. Weakness: the max_a′ over actions is the problem — with millions of items it is intractable, so you restrict A to a retrieved candidate set, or use ANN to approximate the max. Double/Dueling DQN address Q-value over-estimation (which on a long horizon compounds badly).

Scaling the action space — making the max tractable over millions of items

"Restrict A to a candidate set" is the slogan; here are the concrete mechanisms, in the order of how aggressively they cut the action set. The failure mode they all dodge is the same: a single forward pass that scores every item is both too slow for the serving budget and statistically hopeless — you cannot learn a reliable Q for an item with three logged impressions.

Cascading / coarse-to-fine RL. Two staged policies: a cheap coarse-ranker cuts ~10⁷ items to a few hundred, a fine-ranker runs the expensive Q-net only on those. The max is now over ~10²–10³, not 10⁷ — a ~10⁴–10⁵× reduction in the scoring work, mirroring the retrieval → ranking funnel you already build.
Hierarchical category-then-item action. Factor the action a = (c, i): a high-level policy picks a category / direction c (≈10²–10³ options), a low-level policy picks the item i within it. The combined branching is |C| + |I_c| instead of |C|·|I_c| — e.g. 500 categories × 20k items each is 10M flat, but only 500 + 20 000 ≈ 20.5k evaluations factored. This is the same options framework as the HRL section, used here purely to shrink the action branching.
PQ / ANN action layer. Replace the DQN's fully-connected output head (one logit per item — impossible at 10⁷) with a learned query vector and a product-quantization ANN index over item embeddings: the "argmax over Q" becomes an approximate-nearest-neighbour lookup, O(log N) instead of O(N). The net emits a state-conditioned query; the index returns the top candidates as the action set.
Embedding-basis actions (cross-item generalization). Define the action over the item-embedding space — the policy outputs a point/weight vector in embedding space and you retrieve the nearest items — rather than over raw item IDs. A brand-new item with no clicks inherits a Q estimate from its neighbours' embeddings, so you are not learning 10⁷ independent per-ID values; this is what makes the value function generalize to the long tail instead of collapsing on it.

Training at PB scale — distributed actor-learner (Ape-X)

On-policy DQN training — one process that alternates "collect a transition, take a gradient step" — cannot keep a GPU learner fed when the experience source is petabytes of logged interactions. The standard decomposition is Ape-X: many distributed actor workers asynchronously generate experience (replaying logs or running the behavior policy) and push transitions into a shared replay buffer; a single central learner samples from that buffer and does all the gradient updates, periodically broadcasting fresh weights back to the actors. Decoupling the two means you scale data throughput (add actors) independently of update throughput (the learner's GPU), and the actors never block on the learner.

Reward-stratified (prioritized) replay. Positive reward is rare — a uniform sample of transitions is almost all r=0, so the gradient is dominated by "nothing happened." Stratify the buffer by reward and up-sample the high-value transitions. Worked feel: if likes occur on 1% of impressions, uniform sampling gives a 256-batch ≈ 2–3 positive transitions; stratifying to a 30% positive mix gives ≈ 77 — a ~30× denser learning signal per batch for the event you actually care about. (You then importance-correct for the over-sampling, exactly as prioritized replay does.)
Online / offline refresh cadence. Two clocks: refresh the slow bootstrap target network hourly (stability — it must not chase its own estimate intra-day), and the serving online network daily (freshness — yesterday's interactions folded in). Splitting the cadence is how you get adaptivity without the oscillation a single fast clock would cause.
Degrade-to-GBDT fallback. The DRL policy shares the request path with a hard SLA. Wrap its inference in a timeout: if the Q-net does not return within budget (a tail-latency spike, a cold cache), fall back to a lightweight LR / GBDT ranker for that request. You trade a sliver of quality on the slow tail for never blowing the latency contract — and the fallback ranker is the same one you already run as the supervised baseline.

Policy-based: REINFORCE / policy gradient

Skip Q; optimize the policy π_θ(a|s) directly by gradient ascent on expected return J(θ) = E_{π_θ}[G]. The policy gradient theorem gives a usable estimator:

∇_θ J(θ) = E_{π_θ}[ ∇_θ log π_θ(a|s) · ( G_t − b(s) ) ]

Read it as: push up the log-probability of actions that led to better-than-baseline return, push down the rest. The baseline b(s) (typically a learned value V(s)) subtracts the "expected" return so you reinforce only the surprise — this slashes variance without adding bias. Strength: outputs an action distribution directly, so it fits a softmax over a dynamic, huge item set far more naturally than a max, handles continuous actions (fusion weights), and optimizes multi-objective returns end-to-end. Weakness: Monte-Carlo returns make REINFORCE high-variance and sample-hungry, and it is on-policy by default — a serious problem when your data is logged.

Actor-critic: A2C, PPO, DDPG

Combine both: an actor π_θ proposes actions, a critic V_φ (or Q_φ) estimates value and supplies the baseline. Replace the noisy return with the advantage A(s,a) = Q(s,a) − V(s) — "how much better than average is this action?" — usually via GAE, a bias/variance dial between one-step TD and full Monte-Carlo. PPO adds a clipped objective that forbids large policy jumps per update — directly relevant to recommendation, where one bad policy step ships to millions of users before your next A/B readout. This is the workhorse for production recommendation RL.

SAC — maximum-entropy actor-critic

SAC (Soft Actor-Critic) is an off-policy actor-critic that augments the reward with an entropy bonus α·H(π), so the policy maximizes return and stays as stochastic as it can afford to. The objective becomes J(π) = Σ_t E[ r_t + α·H(π(·|s_t)) ]. Two properties make this a natural recsys fit. First, sample efficiency: like DQN it learns off-policy from a replay buffer, reusing logged transitions — precious in a system where every interaction is already recorded and fresh online exploration is expensive. Second, built-in exploration: the entropy term keeps the recommender from collapsing onto a narrow set of items, making exploration a first-class objective rather than a bolted-on ε — the same instinct that motivated the bandits above, but inside the full-horizon policy.

Hierarchical RL — decisions at two timescales

Hierarchical RL (HRL) factors the policy by timescale. A high-level policy (the manager) picks a sub-goal or option — "explore a new interest", "exploit known taste", or a channel/category — and a low-level policy (the worker) picks the concrete item to fill it. The options framework gives this temporal abstraction: the manager acts once per several worker steps. For recommendation this maps cleanly onto the session-level vs item-level split and eases long-horizon credit assignment — the manager optimizes session/retention reward over the slow timescale, the worker optimizes within-session engagement, so the delayed retention signal no longer has to propagate through every single item decision.

	Contextual bandit	Full RL (DQN / PG / A-C)
Horizon	one step (γ=0)	multi-step discounted return
Models actions affecting future state?	no	yes — the whole point
What it estimates	E[r \| x,a] + uncertainty	Q^π, V, or π over the horizon
Sample efficiency	high	low — needs many interactions
Best fit	cold start, exploration, slate where order is independent	session optimization, retention, fatigue, sequencing
Main risk	under-models long-term effects	instability, credit assignment, off-policy bias

Rule of thumb: if showing an item barely changes what you would want to show next, a contextual bandit captures ~all the value at a fraction of the complexity. Reach for full RL only when sequencing and long-term state genuinely matter — and budget for the engineering it costs.

The off-policy problem — learning from logged data

Here is the hard truth of recommendation RL. Policy gradient and on-policy methods assume your data was generated by the current policy π_θ. But you cannot let an untrained agent explore freely on real users — that is lost revenue and bad experiences. Almost all your data is logged: it was collected by yesterday's production policy β (the "behavior policy"), and you want to evaluate or improve a new policy π. The distributions do not match. Naively summing logged rewards estimates β's value, not π's.

Why a correction is mathematically necessary

You want E_a∼π[ f(a) ] but you only have samples a∼β. Importance sampling reweights each logged sample by how much more (or less) the target policy would have taken that action:

E_a∼π[ f(a) ] = E_a∼β[ ( π(a|s) / β(a|s) ) · f(a) ]

the ratio w = π(a|s) / β(a|s) is the importance weight. An action the new policy loves but the old one rarely showed gets up-weighted; one the old policy over-served gets down-weighted. This is exactly the inverse-propensity scoring (IPS) you met in bias & debiasing — the propensity β(a|s) is the logging probability, and you must log it at serving time, because you cannot recover it later.

Worked numbers — the importance weight, and where it explodes

A logged session shows an item the old policy picked with probability β(a|s) = 0.2; the new policy would pick it with π(a|s) = 0.4. Weight w = 0.4 / 0.2 = 2 — this logged transition's reward counts double, because π would have produced it twice as often. Reasonable. Now an item β almost never showed, β = 0.001, that π loves, π = 0.5: weight w = 500. One lucky logged reward of +1 becomes +500 in the estimate, and your whole policy evaluation now hinges on a single noisy sample — the variance is enormous. Clip at w ≤ 10 and that transition contributes +10 instead of +500: you've introduced a downward bias (you under-credit a genuinely good action), but you've cut this sample's leverage 50× (500 → 10) — and since its variance contribution scales with the square of the weight, that term's variance drops by ~50² = 2500×. That bias/variance trade is the entire craft of off-policy correction — and it's why a low-support new policy must lean on batch-RL pessimism rather than pretend the correction is trustworthy.

The catch with importance weights is variance: if π wants an action β almost never took, w blows up and a single sample dominates the estimate. The standard mitigations:

Weight clipping / capping — cap w at some c (e.g. the top-K off-policy correction in YouTube's REINFORCE recommender). Trades a little bias for a lot of variance reduction.
Doubly-robust (DR) estimators — combine a learned reward model with IPS: if either the model or the propensities are accurate, the estimate is unbiased. The standard for off-policy evaluation.
Offline / batch RL (BCQ, CQL, batch-constrained methods) — add a pessimism penalty so the policy cannot claim high value for state-action pairs the log barely covers. This is the deeper fix for the "the data does not support that action" failure: do not extrapolate where you have no evidence.

Off-policy evaluation is not a substitute for A/B

IPS/DR let you screen candidate policies offline cheaply and safely — but they are high-variance and assume the logging policy had support everywhere the new policy wants to go (no support → no correction is possible, only pessimism). Under non-stationarity (interest drift), offline estimates and online reality diverge further. The verdict still comes from an online A/B test. RL changes what you ship and how you pre-screen it; it does not change who decides.

Non-stationarity — interest drift and the safe online update

The MDP table called the transition P(s′|s,a) "non-stationary" and moved on. That word hides the single most destabilizing fact about a live recommender: the user's interests are themselves drifting, on two clocks at once. A breaking-news spike or a one-off shopping trip is a fast, possibly transient swing; a genuine shift in taste is slow. A model that treats a recent click burst as a permanent preference over-reacts to noise; one that smooths everything misses the real shift. Plain discounting does not address this — γ weights future reward, not how recent the evidence behind the current state is.

Dual interest channels + time-decay

The standard structure factors user interest into two channels updated at two speeds, then fuses them as the state:

Short-term channel — sliding-window statistics over the last hour / last N interactions (e.g. category distribution of recent clicks), updated fast and online. Captures the transient swing.
Long-term channel — a slow-updating memory network (a DKVMN-style key-value memory, or a periodically-refreshed long-interest embedding) that captures stable taste and resists being overwritten by a single burst.

Inside each channel, weight behaviors by an exponential time-decay e^{−λt} so recent actions count more. Worked numbers: pick the decay so behavior is "half-forgotten" after one day — solve e^{−λ·24h} = 0.5 ⇒ λ = ln2 / 24 ≈ 0.0289 per hour. Then a click from 6 h ago carries weight e^{−0.0289·6} ≈ 0.84; from 48 h ago, e^{−0.0289·48} ≈ 0.25; from a week ago, e^{−0.0289·168} ≈ 0.008 — effectively gone. Drop λ (slower decay) and the long-term channel dominates; raise it and you track the short-term swing. The two channels are just two values of λ made explicit and learned separately.

Detecting drift, and updating without blowing up

Two more mechanisms make this safe in production:

Drift detection by distribution divergence. Track the JS / KL divergence between the user's recent category distribution and their historical one; a spike flags a genuine interest shift (vs. noise) and can trigger more exploration or a memory refresh. Same tool you would use to monitor feature drift — here pointed at the user's own behavior distribution.
KL-bounded safe online update. A recommender often does incremental updates every few minutes on fresh clicks. The danger: one bad mini-batch ships a degenerate policy to millions before the next A/B readout. Bound each update by a KL-divergence constraint on the policy change, D_KL(π_new ‖ π_old) ≤ δ — the same trust-region instinct as the PPO clip, applied to the online refresh cadence. Worked feel: with a small δ ≈ 0.01, a single 5-minute update can shift the action distribution by at most a few percent, so a corrupted batch degrades gracefully instead of catastrophically; you bound the blast radius of every incremental step rather than trusting each one. Periodically resetting the target network keeps the value estimates from ossifying around stale interests.

Why this is a safe-RL guardrail, not a tuning knob

Interest drift is where the off-policy correction and the reward design both get harder at once: the logging policy's support moves out from under you, and a reward that was well-calibrated last week mis-weights this week. The dual-channel state, the time-decay, the divergence monitor, and the KL bound are not accuracy tweaks — they are the safety envelope that lets a model learn online from a moving distribution without one bad step running wild on live traffic. This is the same control discipline as the pacing / trust-region machinery in 10 · Bidding & pacing, and it is what makes "update every 5 minutes" survivable.

Reward design — where most of the work (and danger) lives

The algorithm is the easy part. The reward function is the product decision, encoded as a scalar, and a strong optimizer will do exactly what you wrote — including the parts you did not mean. Four problems, each with a standard handle:

Problem	What goes wrong	Standard handle
Sparse reward	Follows / purchases are rare → almost all transitions have r=0, nothing to learn from.	Reward shaping: layered proxies (impression → click → completion → like), or inverse RL to infer a denser latent reward from engaged trajectories.
Delayed reward	Retention is observed tomorrow; credit must flow back to today's actions.	Discounted return + bootstrapping (TD / n-step); episode-terminal reward for a return visit; survival models for the delay distribution.
Proxy gaming	Optimizing watch-time breeds clickbait / outrage / autoplay traps; a filter bubble forms.	Multi-objective reward (mix watch, like, share, and diversity/novelty); penalty terms for fatigue, repetition, "skip-after-2s"; hard guardrail constraints.
Short vs long term	Immediate engagement and retention conflict — the central tension of this lesson.	Make retention an explicit (terminal, delayed) reward term; tune γ upward; treat it as a multi-objective problem with retention guarded.

Delayed feedback is not the same problem as discounting — and γ does not solve it

Discounting answers "how much should a reward I will observe at t+k count?" Delayed feedback is a different question: at training time the reward simply has not arrived yet, and a not-yet-observed conversion is indistinguishable from a true negative. A purchase might land 6 hours or 3 days after the impression; if you label every unconverted sample r=0 and train now, you are teaching the model that good recommendations failed. The clean framing is a POMDP: the true reward is a hidden state, and the elapsed time is the only observation you have about it.

Delayed-feedback modelling (DFM) models the delay distribution explicitly. Fit a survival / hazard model (or a self-exciting Hawkes process for bursty repeat behavior) to historical action→feedback gaps, giving S(t) = P(conversion has not yet occurred by elapsed time t | it will eventually convert). Then weight an as-yet-unconverted sample by its probability of still being a true negative rather than calling it a hard 0.

Worked numbers — time-decayed weight on an unconverted sample

Say the fitted delay is exponential with mean 12 h, so S(t) = e^(−t/12) (hours) is the probability a true converter is still silent at elapsed time t. An impression that converted historically has on average a 12 h gap. The weight we want on an unconverted sample is its probability of being a genuine negative, 1 − S(t) — small when the sample is probably just early, growing toward 1 as the silence drags on. Take a sample shown t = 2 h ago with no conversion yet: S(2) = e^(−2/12) = e^(−0.167) ≈ 0.85, so it is ~85% likely to just be early — weak evidence of a true negative, so weight it only 1 − S(2) ≈ 0.15, not a hard 0. The same sample re-examined at t = 24 h with still no conversion: S(24) = e^(−2.0) ≈ 0.135 — now only ~14% likely to still be early, so it earns a near-full negative weight 1 − S(24) ≈ 0.865. The label "matures" from "probably just early" toward "almost certainly a true negative" as elapsed time grows. This is what discounting alone cannot express — γ scales a reward you have seen; DFM tells you how much to trust the absence of one. A cheaper variant, the pseudo-immediate reward, replaces the slow signal with a short-term proxy strongly correlated with it (click as a stand-in for next-day return) — but only after you have validated that correlation offline; trust an uncorrelated proxy and you are back to optimizing the wrong thing.

A typical composite immediate reward, with a long-term term folded in:

r = α·(watch-time / video-len) + β·𝟙[like] + δ·𝟙[share] − λ·(fatigue / repetition) + μ·𝟙[returns tomorrow]

The weights are not free parameters to tune for offline reward — they are business policy, validated by online guardrail metrics. Set λ=0 and you will ship the engagement trap; set μ too low and you optimize a great Tuesday at the cost of next month.

Multi-objective reward mechanics — the part the formula hides

Writing R = 0.7·click + 0.2·watch + 0.1·share looks like you have set the priorities to 70/20/10. You have not — unless every term is on the same scale. This is the most common way a multi-objective reward silently mis-weights itself, and three mechanics fix the three failure modes.

Worked numbers — why you must scale-normalize before weighting

Raw signals live on wildly different scales. Suppose, per impression: click ∈ {0,1} (mean ≈ 0.10), watch-time in seconds ranges 0–60 (mean ≈ 18), and share ∈ {0,1} (mean ≈ 0.01). Plug the raw values into R = 0.7·click + 0.2·watch + 0.1·share and the typical contributions are:

term weight raw mean contribution = weight × raw mean click 0.7 0.10 0.070 watch (sec) 0.2 18.0 3.600 ← dominates everything share 0.1 0.01 0.001 ───── watch is ~51× the click term and ~3600× the share term

The "0.2" weight on watch-time is a fiction: in practice watch-time is ~98% of the reward, and the policy will chase seconds and ignore clicks and shares entirely. The fix is to scale-normalize each objective to a common unit first — Z-score (x−μ)/σ, or map each to [0,1] — so a weight of 0.7 actually buys 70% of the gradient. After Z-scoring, each term has mean 0 and unit variance, and the 70/20/10 split is the real split. Always normalize, then weight. The two production reward vectors the literature reports — R = 0.7·click + 0.2·watch_norm + 0.1·share and R = 0.7·progress + 0.2·like + 0.1·share — both carry the _norm / progress (a ratio in [0,1]) precisely because the raw second-count cannot be mixed in directly.

Once the scales are honest, two more guardrails keep the optimizer from sacrificing a secondary objective to win the primary one:

Weight floors. Pin a minimum weight on the objectives that protect the experience — e.g. keep the diversity / CTR-style weight ≥ 0.3 — so even when the optimizer would happily zero them out to maximize watch-time, it structurally cannot. A floor is a cheap, legible defence against the engagement trap.
Pareto hard-constraint. Reframe the secondary metric as a constraint, not a soft term: "maximize completion-rate subject to CTR-drop ≤ 5%." Any candidate policy that breaches the constraint is rejected outright regardless of how good its primary metric looks — this is what stops a Pareto-dominated "win" from shipping. It maps directly onto the A/B guardrail: CTR is a launch gate, not a term you can trade away.
GradNorm / uncertainty weighting. Hand-tuned static weights drift out of date as objectives train at different rates — the term whose gradient is already small gets effectively ignored. GradNorm rebalances the weights dynamically so each objective's gradient magnitude stays comparable, and uncertainty weighting (homoscedastic task uncertainty) does the same from a Bayesian angle. Use these when static weights keep needing manual re-tuning across launches.

This is the same multi-task balancing problem you meet in multi-task ranking and again in 25 · Multi-agent RL, where each objective can be a separate agent and the conflict is resolved by reward shaping rather than a single scalar.

Interactive · immediate vs long-term reward

A toy two-policy world. The "clickbait" policy wins more immediate engagement per session but raises churn; the "balanced" policy earns a bit less now and keeps users longer. Slide the discount γ and the churn penalty μ and watch which policy actually wins on lifetime value — the number that should decide what you ship.

The practical recipe

Putting it together, the order you would actually build this in:

Don't start with RL. A well-built multi-task supervised ranker with watch-time and retention predictions in the score captures most long-term value at a fraction of the cost. RL is the increment on top, not the baseline.
Add a contextual bandit for exploration where it is clearly missing — cold-start items and users (15 · Cold start). This is the cheapest, safest slice of the RL framework and often the highest ROI.
Move to full RL only for sequencing/session value. Use a sequence model as the state encoder, an actor-critic (PPO-style, clipped) for the policy, restrict actions to the retrieved candidate set, and design a multi-objective reward with explicit fatigue and retention terms.
Learn off-policy from logs. Log propensities at serving; correct with capped importance weights; pre-screen with doubly-robust off-policy evaluation; constrain with batch-RL pessimism so you never extrapolate into unsupported actions.
Ship behind an A/B test, with guardrails. Monitor not just the reward but diversity, fatigue, and retention. Constrain per-update policy change (PPO clip) so a bad step cannot run wild on live traffic before you measure it.

Serving RL under a hard latency budget

Everything above is about learning a good policy. None of it ships if it cannot answer in time. This is the constraint that quietly kills naive RL recommenders, and it is worth stating numerically: the ranking critic must score in ~10ms, end-to-end recommendation serve in <80ms, and live-streaming — even more latency-sensitive than short-video — also <80ms with far less slack. A heavy deep-RL policy with a Transformer state encoder, a stochastic actor that samples a distribution, and a Q-net scoring thousands of candidates does not fit in that budget. The whole serving problem is reconciling "the policy we trained" with "the time we have."

The standard fix — a two-model split: heavy offline, light online

Do not serve the deep-RL model. Run a dual-model architecture: a heavy offline DRL policy (full state encoder, expensive critic, off-policy / batch-RL training) learns the values; a lightweight online model (a small DNN, often with a linear RL head) actually serves real-time requests. The offline model periodically distills its knowledge into the online one — the online net is trained to match the offline policy's action distribution / Q-ranking, so it inherits the long-horizon behavior at a fraction of the inference cost. The expensive reasoning happens offline on the slow clock; the request path runs only the distilled student. This is the recommendation-RL version of teacher→student distillation, and it is what makes a sub-10ms response with deep-RL quality possible at all.

Two more cuts that get you the rest of the way

State-statistic simplification (~70% feature-compute cut). Replace the raw behavior sequence (which needs an RNN/Transformer pass per request) with cheap sliding-window statistics — e.g. "mean CTR / mean watch-% over the last 10 interactions." You lose some fidelity in the state, but pre-computing these aggregates (Flink, ahead of the request) removes the per-request sequence encode and cuts roughly 70% of feature-computation time. The Markov state is now a short stat vector, not an encoded history.
Action-cluster compression. Cluster the ~10⁷-item library into a few hundred clusters and have the online model predict only cluster-level Q; resolve the concrete item within the chosen cluster with a cheap secondary step. Scoring ~10² clusters instead of 10⁷ items is the difference between fitting the budget and not — and it composes with the cascading / PQ-ANN action layers from the DQN section.

Round it out with the systems layer: TensorRT-compiled model instances, GPU-cached embeddings for hot users/items, and a Kafka-decoupled async design where the online service only does action prediction while model updates are consumed offline. And know the trade curve: dropping latency 500ms → 200ms may cost <1% on the metric, but pushing 200ms → 50ms can cost ~5% — so you locate the operating point on a latency-vs-quality Pareto front, not at the lowest latency you can technically hit.

Interview prompts you should be ready for

"Why is predicting the next click not enough — when do you actually need RL?" (Supervised ranking is myopic: it optimizes P(click) one independent impression at a time. Two things break that: actions change the user's future state, and the real objective is a discounted return over a horizon, not a point. Reach for RL only when sequencing / fatigue / retention genuinely matter; otherwise a multi-task ranker captures most of it.)
"Map a recommender onto an MDP." (State = user embedding + recent behavior sequence + context; action = item or Top-K slate (or fusion weights); transition = how user interest/fatigue evolves, unknown so go model-free; reward = engagement/retention; γ trades immediate vs future. Note the state is partially observed — strictly a POMDP, and the sequence model is the state estimator.)
"A contextual bandit vs full RL — what is the difference, and which do you start with?" (Bandit is the γ=0 special case: estimate E[r|x,a] plus a principled explore/exploit rule, no modeling of how actions change state. Start with the bandit for cold start and exploration — cheapest, safest, highest ROI. Go full RL only for session/retention sequencing.)
"Write the policy gradient and explain every term." (∇θ J = E[∇θ log πθ(a|s)·(Gₜ − b(s))]. Push up log-prob of actions whose return beat the baseline b(s)=V(s); subtracting the baseline reduces variance without bias by reinforcing only the surprise. Outputs a distribution, so it fits a huge dynamic item set better than DQN's max.)
"Your training data came from last week's policy. Why is that a problem and how do you fix it?" (It is off-policy: naive reward sums estimate the behavior policy β's value, not the target π's. Correct with importance weighting w = π/β = IPS; cap w to control variance; use doubly-robust for evaluation; add batch-RL pessimism (CQL/BCQ) for unsupported actions. Must log propensities at serving — you cannot recover them later.)
"You optimized watch-time and engagement went up but retention dropped. What happened and what do you change?" (Proxy gaming: the optimizer found clickbait/outrage that wins the immediate proxy but exhausts users. Fix the reward, not the model — multi-objective reward with diversity/novelty, fatigue and repetition penalties, and an explicit delayed retention term; raise γ. The weights are business policy, validated by online guardrails.)
"You have a great offline off-policy estimate. Can you ship on it?" (No. OPE screens candidates but is high-variance and assumes the logging policy had support everywhere the new policy goes; non-stationarity widens the gap. The decision still comes from an online A/B test with guardrails. RL changes what you ship and how you pre-screen — not who decides.)
"Your deep-RL ranker is great offline but the critic takes 40ms and serving budget is 10ms. Ship it." (Don't serve the heavy model. Two-model split: heavy offline DRL learns values, a lightweight online DNN+linear-RL head serves, and the offline model distills into it periodically so the student inherits long-horizon behavior cheaply. Then cut state to sliding-window stats (~70% less feature compute, no per-request sequence encode) and compress the action space to ~hundreds of clusters (predict cluster-level Q, not 10⁷ item logits). Add TensorRT + GPU-cached embeddings + Kafka-async updates. Pick the operating point on the latency-vs-quality Pareto front — 500→200ms may cost <1%, 200→50ms ~5%.)
"You wrote R = 0.7·click + 0.2·watch_time_sec + 0.1·share and watch-time still dominates everything. Why, and what else do you add?" (Scales differ: raw watch-time (~18s mean) swamps a 0/1 click — the 0.2 weight is a fiction, watch is ~98% of the reward. Scale-normalize each objective (Z-score / [0,1]) before weighting so the weights mean what they say. Then weight floors (keep diversity/CTR weight ≥0.3 so the optimizer can't zero it), a Pareto hard-constraint (maximize completion s.t. CTR-drop ≤5% — a launch gate, not a tradeable term), and GradNorm / uncertainty weighting to rebalance dynamically instead of re-tuning by hand.)
"User interests drift and you update the policy every 5 minutes. How do you stay adaptive without a bad update tanking the feed?" (Dual interest channels — short-term sliding-window stats updated fast, long-term DKVMN-style memory updated slow — fused as the state, with e^(−λt) time-decay inside each (λ = ln2/24h ≈ 0.029/h for a one-day half-life). Detect real drift via JS/KL divergence of the user's category distribution. Bound every incremental update with a KL constraint D_KL(π_new‖π_old) ≤ δ — a trust region on the online refresh that caps the blast radius of any one batch — and periodically reset the target network so values don't ossify.)

Takeaway

Supervised ranking optimizes the next click; RL optimizes the discounted return over the horizon, acknowledging that today's recommendation changes tomorrow's user. The framework is clean — an MDP plus the Bellman equation, with bandits as the γ=0 corner — but the leverage and the danger both live in the parts that are not equations: what you choose to reward, and how honestly you correct for the fact that your data came from a different policy than the one you want to ship.