Real-time & streaming recommendation

A recommender is a function of features, and features go stale. The model that ranked your feed at 9:00 does not know that, at 9:02, you watched three cooking videos, that a story just broke, or that the clip everyone is sharing was uploaded ninety seconds ago. "Real-time" is the discipline of closing that gap — treating freshness as a feature and building the streaming machinery to deliver it in milliseconds.

Why freshness is itself a feature

Take the same user and the same item at two different moments. The only thing that changed is time, yet the right recommendation flipped. That is the operational definition of freshness-as-signal: a feature whose value decays, often steeply. It is not a system-health nicety — it is predictive signal that routinely dominates the static features you painstakingly engineered. Three canonical sources of perishable signal:

• Intra-session intent — the user's last 5–10 interactions. Someone who just skipped three dance clips and finished two recipes has told you more in 30 seconds than their 6-month profile says. Skips and fast swipes (negative feedback) are more perishable and often more informative than likes.
• Item-side trends — a clip's live completion and like rate. New content has a short life (a short video often decays within 24h), so its early engagement is the only signal you get; react in minutes or the window closes.
• Context & world state — breaking news, live sports, a meme spiking, time-of-day. None of this exists in yesterday's training data.

We can make "freshness is a feature" literal. Model the usefulness of a signal observed Δt ago as exponential decay, and you recover the half-life intuition behind every freshness SLA:

value(Δt) = value₀ · e^−λ·Δt  |  half-life t½ = ln 2 / λ ≈ 0.693 / λ

If a click's signal has a half-life of 2 minutes (λ ≈ 0.0058/s), a feature pipeline with a 4-minute lag delivers it at e^{−0.0058·240} ≈ 0.25 — you threw away 75% of its value before the model saw it. A freshness SLA is just "keep e^−λ·lag close to 1." Different signals have wildly different λ, which is exactly why a single update cadence is wrong, and why we split the work across planes.

The three planes

You cannot do all the work at request time (too slow), and you cannot do it all in batch (too stale). The resolution is to split the system by how fresh the work must be into three planes, each with its own latency budget and its own job. Almost every later design decision reduces to "which plane does this belong on?"

Three inference topologies — and when each fits

The three planes tell you where a feature lives. A separate, easily-conflated question is where the scoring happens: when a request lands, do you compute the model output synchronously, look up a precomputed one, or hand the work to a queue and poll for it later? There are three topologies, and they trade the same currency — freshness against latency against blast radius. The mistake is picking one for the whole system; the senior move is picking one per funnel stage.

Worked latency, same ranker, three topologies. Say the heavy model is 35ms of compute, a feature-store fetch is 10ms, and a light LR correction is 3ms. (a) spends 10 + 35 = 45ms synchronously and inherits the full variance of both — a feature-store P99 of 40ms drags the request P99 with it. (b) spends 10 + 3 = 13ms online (the 35ms ran offline, off the clock), comfortably under the <50ms line and almost flat because there is no heavy compute left to spike. (c) returns in milliseconds (just "accepted, poll later") but the answer arrives ~1–3s later when Flink drains the queue — fine for a feed refresh, fatal for the first screen.

The streaming substrate — log + processor

The nearline plane is built on two pieces that recur in every design: an append-only log (Kafka) and a stream processor (Flink, or Kafka Streams / Spark Structured Streaming).

Kafka — the log as the source of truth

Every behavioral event — exposure, click, watch-time, skip — is appended to a partitioned, durable log. Two design knobs matter:

• Topics by SLA, not by convenience. Separate the high-volume exposure firehose from low-volume, high-value interaction events (likes, shares) so a flood of one cannot starve the other. Isolate the top ~1% hot items into their own topic so a viral clip doesn't skew a single partition.
• Partition by key for order, round-robin for throughput. Hash by user_id when a user's events must stay ordered (you're building a session sequence); spread randomly for order-agnostic counters. Rule of thumb: partitions ≈ 2× the downstream processor's max parallelism, with ~20% headroom for spikes.

Flink — stateful, event-time stream processing

The processor consumes the log and maintains keyed state — e.g. a per-user MapState of interest tags with time-decayed weights, or a per-item rolling completion rate over a sliding window (say 1h window, 5s slide). It keys by user or item, processes each event, and writes the updated feature to the online store. State lives in a RocksDB backend with incremental checkpoints and TTL eviction so it doesn't grow unbounded. Crucially it uses event time + watermarks, not wall-clock, so out-of-order and late events still land in the right window — the difference between a feature that is correct and one that silently drifts.

Exactly-once vs at-least-once — and what each costs

The log can deliver an event to your processor in three ways, and the choice is a direct trade between correctness and throughput:

Flink achieves exactly-once with a checkpoint + two-phase-commit mechanism. The JobManager periodically injects a barrier into the stream; operators snapshot their state (Chandy–Lamport distributed snapshot) when the barrier passes. End-to-end correctness needs the sink to participate: a TwoPhaseCommitSinkFunction writes a transaction but only commits when the checkpoint completes globally — so a crash rewinds to the last consistent snapshot with no partial output. Pair it with an idempotent Kafka producer (enable.idempotence=true) and transactional reads.

The real-time feature store & freshness SLAs

The nearline plane writes; the online plane reads. The contract between them is the online feature store — a low-latency KV layer (Redis, or a purpose-built store) holding the freshest value of each feature, keyed by entity. Three properties make or break it:

• Single feature definition. A feature must be computed by one piece of code and read identically at training and serving time. Two definitions is the canonical source of train–serve skew (covered next lesson). This is why the field is migrating from Lambda (parallel batch + streaming paths computing the same feature twice) to Kappa (everything is a stream; a "batch" recompute is just replaying the log from offset zero). One pipeline, one definition, no skew by construction.
• Per-feature freshness SLA. Each feature carries a target lag derived from its half-life — "user session sequence ≤ 2s stale", "item trend stats ≤ 30s", "90-day CTR ≤ 24h". The SLA is measurable: log the event timestamp and the store-write timestamp and alert on the p99 gap.
• A stale fallback. Every feature read must have a last-known-good default, because the online plane cannot block on a slow or missing write. Serving a 30s-old trend beats serving nothing.

Online / incremental learning — and why it's dangerous

The most aggressive form of real-time is updating the model, not just its features. Instead of waiting for the daily offline retrain, you nudge weights continuously from the streaming label feed. The update is plain online SGD on each freshly-labeled example (or mini-batch):

w_t+1 = w_t − η · ∇_w ℓ(w_t; x_t, y_t)

FTRL-Proximal — the workhorse for sparse linear CTR

Plain online SGD on a logistic CTR model adapts fast but produces dense weights: with billions of hashed/cross features, every coordinate it ever touches stays nonzero, so the served model never fits in memory. Sparsity here is not a nicety — it is a serving requirement. FTRL-Proximal (McMahan et al., 2013, the Google ad-click paper) is the canonical answer: it matches SGD's online convergence and yields exact zeros via an L1 term, which is why it became the default for high-cardinality linear CTR models on the online/nearline plane.

The intuition: instead of one gradient step, FTRL keeps a running sum of past gradients and, each step, solves a regularized objective — "stay near the average of past gradients" plus L1 (the term that induces zeros) and L2. Crucially the learning rate is per-coordinate and adaptive: a feature seen rarely accumulates little and gets a larger effective step, so cold features still learn. The per-coordinate update has a closed form that snaps a weight to exactly 0 whenever its accumulated gradient stays under the L1 threshold: |z_i| ≤ λ₁ ⇒ w_i = 0, otherwise w_i = −(z_i − sgn(z_i)·λ₁) / ((β + √n_i)/α + λ₂). That single branch is what gives you a model that is both incrementally fresh and sparse enough to serve.

It lives in this lesson because it is the workhorse for the online/nearline plane's incremental updates on high-cardinality sparse features, pairing directly with the streaming feature pipeline that feeds it labeled events. But it earns no exemption from the rest of the discipline: an FTRL model updated from the live label feed needs the same train–serve-skew and drift guards the next lesson emphasizes, plus everything below.

This is how a model adapts to a breaking trend within minutes rather than a day. It is also a loaded gun, for reasons that are structural, not incidental:

• Feedback loops. The model's own outputs generate the data it next trains on. Over-recommend a video → it gets more clicks → it looks even better → recommend it more. Online learning tightens this loop from a day to minutes, accelerating popularity bias and the Matthew effect. Inverse-propensity weighting (IPS) and forced exploration are the antidotes.
• Instability. A single bad batch — a logging bug, a bot attack, a hot item — can poison weights in real time, with no overnight human checkpoint to catch it. You need gradient clipping, learning-rate caps, anomaly detection on the loss, and the ability to instantly roll back to the last good checkpoint.
• Delayed & biased labels. "Did the user finish this video?" arrives seconds-to-minutes late; "did they retain for 7 days?" never arrives in time for an online update at all. Training on whatever labels have landed so far biases toward fast signals (clicks) over slow value (retention).

Operating the loop — the machinery that keeps it alive

The dangers above are not arguments against online learning; they are the spec for the operational scaffolding it needs. A production loop has four mechanisms, each answering one of the failure modes directly.

1 · Tiered drift detection → graded action. Watch the input distribution with a Population Stability Index on a sliding window (per-segment baselines, since "new users" and "power users" drift differently), with a KS test (continuous features) or chi-square (categorical) to confirm the shift is significant rather than sampling noise. The response is graded by severity, not a single tripwire — over-reacting to noise is as costly as missing a real shift:

Recall PSI is Σ (pᵢ − qᵢ) · ln(pᵢ / qᵢ) over bins, where p is the live distribution and q the training baseline. A feature that moves about 5 percentage points of mass from one decile (0.10 → 0.05) into an adjacent one (0.10 → 0.15) lands around PSI ≈ 0.055 — squarely in the "watch" band, the no-op case you want the threshold to absorb so the loud bands stay meaningful. Push that to an 8-point shift (0.10 → 0.02) and PSI jumps to ≈ 0.18 — already in the alert band, because the emptying bin's relative swing ln(0.02/0.10) dominates the sum. That sensitivity to nearly-empty bins is exactly why the raw threshold needs the metric-drop conjunction above it.

2 · Swap models by interpolation, not by flip. An online learner produces a new weight vector continuously, but slamming θ_old → θ_new in one step makes the ranking lurch — yesterday's feed and this second's feed disagree, and the user sees the seam. Ramp it instead, blending the two parameter sets and walking the mix:

θ = α·θ_new + (1 − α)·θ_old,   α: 0 → 1 over the ramp

At α = 0.1 the served model is 90% the trusted old weights and 10% the new — a 10% nudge, not a jump — and you step α up (say 0.1 → 0.3 → 0.6 → 1.0) only while guardrail metrics hold. If a step degrades them, you walk α back down; the rollback is continuous, not a cliff. (This is the parameter-space cousin of the traffic-ramp canary in 18 · Deployment — there you ramp the fraction of users; here you ramp the weights themselves.)

3 · Sync to the parameter server safely. The online learner doesn't serve directly — it pushes updated parameters to the serving parameter server (PS) on a cadence, typically ~5 minutes, not per-example (per-example sync would hammer the PS and couple serving latency to training noise). Two guards on every sync: a double-buffer so reads never block on the write (serve buffer A, write buffer B, atomic pointer flip — the embedding-table buffering engineering is canonical in 18 · Deployment; don't re-derive it), and a dimension-consistency check before the flip — if a feature was added or the hash space resized, the new parameter block's shape won't match the serving graph's expectation, and swapping it in would silently corrupt every score. Check shape, then flip; mismatch aborts the swap and keeps the old buffer live.

4 · Periodically retrain from scratch. An online loop trained only on its own freshly-served traffic drifts into a local optimum: it keeps exploiting the slice of the catalog it already favors, the feedback loop ossifies those weights, and it forgets how to value anything it stopped showing. The escape is a periodic full retrain on the broad historical log (the offline plane's job, T+1 or weekly) that re-anchors the weights on the whole distribution, then hands a fresh, unbiased checkpoint back to the online loop to continue from. Online learning tracks the fast-moving present; the full retrain stops it from confusing "what I've been showing" with "what is good."

The latency budget — arithmetic, not vibes

"Real-time" means the online plane returns inside the user's tolerance. A typical product budget is a P99 of ~200ms end-to-end, of which network round-trip and rendering already eat a chunk. The recommender owns maybe 120–150ms of server time, and it must spend that across the whole funnel in series. The budget is additive — a literal subtraction problem:

T_server = T_{feature-fetch} + T_retrieval + T_ranking + T_rerank + T_overhead ≤ ~150ms

Every stage you add, every feature you fetch, every extra ANN probe is drawn from one shared account. This is why the funnel exists: you spend cheap milliseconds over millions of items in retrieval and expensive milliseconds over hundreds in ranking. It is also why "let's add a 20ms bandit re-rank" is a budget decision, not a free lunch — the same arithmetic resurfaces in 18 · Deployment.

Interactive · freshness-decay & latency-budget

Two tools. The first turns a half-life and a pipeline lag into the fraction of signal that survives to inference — the number a freshness SLA defends. The second distributes a server budget across the funnel; red means you blew the P99 and something must get cheaper or move to the nearline plane. The numbers are illustrative; the trade-offs are real.

Interview prompts you should be ready for