Multimodal recommendation

An ID embedding for a brand-new video is a row of noise — no gradient has ever touched it, because nobody has watched the video yet. But the pixels, the audio waveform, and the title text all exist on the first second of upload. Turning that raw content into vectors a recommender can use is the cold-start lifeline.

The problem: ID embeddings are undefined on day zero

A standard recommender represents item i by a learned vector eᵢ ∈ ℝ^d, pulled from an embedding table indexed by item ID — the same sparse table the ranker is mostly made of. That vector is shaped only by interaction signal: every click on i nudges eᵢ toward the users who clicked. The gradient on row i is non-zero only on training rows that contain item i:

∂L/∂eᵢ = 0  whenever item i appears in zero training rows

So a video uploaded one minute ago has an embedding equal to its random initialization — it sits nowhere meaningful, and the ranker scores it essentially at random. This is the item cold-start problem in its purest form, and no ranking architecture fixes it, because the input itself is uninformative.

A content embedding breaks the dependence on interaction history. Instead of learning a free vector per item, we compute one from the item's content with a function f trained on other items (or a giant external corpus):

cᵢ = f(pixelsᵢ, audioᵢ, textᵢ)  — defined on day zero, no clicks required

Two videos of sleeping cats land near each other in c-space because they look alike, not because the same users watched them. A fresh cat video therefore inherits the neighborhood — and the audience — of cat videos the system already understands. That single property is why every modern feed pours engineering into content encoders.

One encoder per modality

Each modality is a different kind of object — a pixel grid, a waveform, a token sequence — so each gets a specialist encoder that maps it into a fixed-length vector. The encoders are almost always pretrained (on ImageNet, on audio tagging, on web text) then optionally fine-tuned, because labeled recommendation data is tiny next to the corpora those backbones saw.

The key abstraction: after encoding, modality differences are erased at the type level — everything is now an ℝ^d vector. Pixels and a title that began as completely different data structures become two vectors that downstream layers can add, concatenate, or attend over. Encoders usually project to a modest d (often 256), because — as the engineering section shows — these vectors get stored and served for every item in the corpus, so width is a real cost.

Picking the backbone — the numbers behind the choice

"Use a ViT" is not an answer; an interviewer wants to know which encoder and why that one. The choice trades three axes against each other — alignment quality (does it match cover to title?), temporal modeling (does it see motion, not just one frame?), and FLOPs (can the upload pipeline keep up?). Concrete anchors, all reported on standard benchmarks:

The text side is rarely just the title. A muted-with-captions clip has its words in the pixels — recover them with OCR on sampled frames; a talking-head clip has its words in the audio — recover them with ASR (speech-to-text), then encode the transcript with BERT. So one "text vector" can be the concatenation of three sources (title + OCR + ASR), and a clip with an empty title is still far from textless.

Frame sampling — you cannot afford every frame

A 30-second clip at 30 FPS is 900 frames; running a ViT over all of them per video, times millions of daily uploads, is a non-starter. The standard cut: sample 1–3 FPS (or detect shot boundaries and keep one frame per shot), center-crop vertical video, then mean/max-pool the per-frame vectors into one clip vector (a Non-local block can replace the pool when you want long-range frame dependence). 1–3 FPS over 30 s is ~30–90 frames instead of 900 — a 10–30× cut in encode cost for almost no loss in content signal, because adjacent frames are nearly identical.

Fusion — where do the modalities meet?

Now we have v (visual), a (audio), t (text), each in ℝ^d. The question is where in the model do they combine? Three answers, in increasing order of interaction richness and cost.

The math makes the tradeoff concrete. Early fusion lets the joint MLP form any cross term — e.g. a hidden unit that fires only on "calm visual × upbeat audio," a contradiction signal a late model can never see. Late fusion can only mix at the very end:

late:  ŷ = σ(wᵥ·sᵥ + wₐ·sₐ + wₜ·sₜ)  — a weighted vote, no v×a term exists

Attention / gating sits between: a gate reads the input and outputs per-modality weights that change example to example — downweight audio late at night; lean on the visual modality for a fast-scrolling user. Same gating idea as MMoE in multi-task learning, repurposed across modalities:

g = softmax(W_g·x)    z = gᵥ·v + gₐ·a + gₜ·t  (gate g depends on the example)

The gate is context-driven, not just per-item

"Downweight audio at night" is one instance of a much bigger idea: the gate's input x is not only the item's content — it carries real-time context and user behavior, so the same clip is fused differently for different users and moments. This is the engineering payload of the gate, and it is worth more than one example:

• Scroll speed. A user fast-scrolling during coarse rank cannot read a title — so the gate leans on the visual modality and downweights text, because only the thumbnail registers at that speed.
• Time-of-day / scene. Late-night or pre-sleep sessions downweight audio (sound is off); a commute session may lean differently than a couch session. Scene is a feature into W_g.
• Content type. Fashion/dance clips are visual-dominant; news/explainer clips are text-dominant. The gate learns this from the content tag, so it does not hand the same weights to a runway video and a headline.
• Behavior history. A user whose history is image-and-text-heavy gets audio downweighted; an electronic-music lover gets it upweighted. The gate reads the user's modality preference.

There is a second, harder reason the gate reads context: load. Under latency pressure or on a low-end phone, the gate can skip an expensive modality entirely — drop the heavy video tower, serve on cached cover + text — and the model still produces a score because Modality Dropout (below) trained it to survive a missing input. So the same gate that decides "which modality matters here" doubles as the lever for "which modality can I afford to compute right now." That is why dynamic, context-aware fusion is strictly more useful than a static weighted sum: it is both an accuracy knob and a serving-cost knob.

What the fusion choice is worth — the A/B numbers

The richer-but-costlier strategies earn real metric wins, and quoting them is how you justify the engineering. Reported figures from production short-video systems:

None of this is free. Adding the multimodal path costs roughly +20% serving latency over an ID-only model, and a full cross-modal Transformer denoiser can add ~300ms if you run it online — which is why nobody does. The discipline that makes mid-fusion shippable is a hard split:

Precompute ~90% of the multimodal features offline; online you run only a sub-50ms fusion head over cached vectors. The ROI rule that decides whether a heavier fusion ships: +50ms of added latency has to buy at least +0.02 AUC, or the win does not justify the serving cost. The industrial sweet spot most teams converge on is gated attention + a residual connection (the Kuaishou recipe) — dynamic per-example modality weighting, but cheap enough to serve, with the residual keeping a clean path for the dominant modality. One subtlety when concatenating heterogeneous vectors: a 2048-d ResNet pool and a 768-d BERT [CLS] have wildly different scales and variances, so apply modality-specific BatchNorm before fusion or the large-norm modality silently dominates the gate.

The fusion landscape, drawn

Cross-modal alignment — the CLIP idea

Encoding each modality separately leaves a problem: the text vector and the image vector live in different coordinate systems. "a golden retriever" (text) and a photo of one (image) are about the same thing but land in unrelated regions. This is the modality gap. To search across modalities — or to make a missing modality recoverable — we need a shared space where matching pairs sit close.

Contrastive learning builds that space. Take a batch of N true (image, text) pairs from the same videos. Encode and L2-normalize to get image vectors u₁…u_N and text vectors w₁…w_N. The matched pair (uᵢ, wᵢ) should be most similar; every other text in the batch is a negative for image i. The InfoNCE loss (CLIP's loss) makes that true:

L = −(1/N) · Σᵢ log [ exp(uᵢ·wᵢ / τ) / Σⱼ exp(uᵢ·wⱼ / τ) ]  (+ the symmetric text→image term)

Read it as an N-way classification: "given image i, pick its caption out of the N captions in the batch." The numerator rewards the true pair; the denominator's sum over j≠i pushes mismatched pairs apart. The temperature τ sharpens (small τ) or softens (large τ) the softmax. After training, cosine similarity in this space means semantic relatedness across modalities — exactly the property we wanted. The denominator's N−1 in-batch negatives are why CLIP trains on enormous batches (32k+): more negatives per anchor sharpen the gradient.

Beyond a flat shared space, richer models (ViLBERT, co-attention) let modalities cross-attend token-by-token — the word "cat" binding to the cat region — catching fine associations at higher cost. Alignment (a shared space) and interaction (cross-attention) are complementary: align first so the spaces are comparable, then attend for fine-grained fusion when the budget allows.

CLIP vs BLIP — alignment vs fusion

CLIP is alignment only. Two independent towers — an image encoder and a text encoder — trained by the InfoNCE loss above to land matching pairs close in one shared space. The towers never talk to each other; the only coupling is the contrastive objective. That independence is exactly the property retrieval wants: because the image side needs no text and the text side needs no image, you can run the image tower over the entire catalog offline, write 256-dim vectors to a feature store, and serve them through an ANN index (03 · Embeddings & ANN) — the same two-tower shape candidate generation already relies on. The weakness is the flip side: a dual encoder only scores similarity. It cannot generate a caption, answer a question about the image, or reason about fine cross-modal detail — there is no path for the pixels and the tokens to interact.

BLIP (and BLIP-2) add a fusion / generation path. On top of contrastive alignment, BLIP wires image–text cross-attention into the encoder and trains generative objectives — captioning and image-grounded text — so the model can produce language conditioned on pixels, not just match the two. BLIP-2 makes this cheap with a Q-Former: a small set of learned query tokens that cross-attend to a frozen vision encoder and feed a frozen LLM, bridging the two without retraining either. The payoff is rich captions, VQA, and fine cross-modal understanding — exactly what you want to describe a brand-new clip. The cost: the fused encoder mixes image and text inside the network, so it is no longer a clean dual tower. You cannot pre-encode a billion items into one ANN-able vector from it; it is a re-rank / understanding tool, not a retrieval index.

The recsys verdict: use CLIP-style dual encoders at the retrieval stage for cold-start content embeddings, where the offline-precompute-then-ANN property is non-negotiable; use BLIP-style models offline to generate captions and content tags for cold items, then feed those generated tags back in as features (or as the text side of the CLIP tower). Aligned-but-separate beats fused the moment you need to pre-index a billion items.

Interactive · the contrastive temperature demo

A batch of N pairs. The cosine of the matched pair versus the average mismatched cosine, run through the InfoNCE softmax. Watch how a tighter matched gap, more negatives, and a smaller temperature drive the matched probability — and the loss — the way training wants.

Putting content vectors to work in the funnel

Content embeddings are not a model — they are inputs, and they plug into two stages of the cascade you already know.

As a retrieval channel. Build an ANN index (03 · Embeddings & ANN) over every item's joint content vector. Now you have a recall source from candidate generation that, unlike collaborative-filtering recall, works for items with zero interactions — the explicit cold-start lifeline. A practical feed runs several recall channels in parallel (hot items, CF neighbors, content-similar, social) and merges; the content channel is the one that carries fresh uploads. The user side can query it by their long-term content-taste vector or, for cross-modal search, by a text query.

As ranking features. Feed the per-modality vectors (and a fused vector) into the ranking model as dense features alongside the ID embeddings. For a warm item the ID embedding dominates and content adds a little; for a cold item the ID embedding is noise and content carries the prediction. The model learns this split on its own — which is why a sensible design is to blend, letting a learned gate decay the content weight as interaction signal accumulates, rather than switching hard between two code paths.

When a modality is missing — and why training must rehearse it

Real catalogs are ragged. A clip uploads with no title; a user mutes the audio; the cover is a black frame; ASR returns nothing for a wordless dance clip. The naive fix — feed a zero vector for the absent modality — is a quiet disaster, and understanding why is the whole point.

Fix part 1 — tell the model it's missing. Add a missing-modality flag bit per modality to the two-tower / ranker input. Now "audio = 0, audio_present = 0" is a distinct, learnable state, not indistinguishable from "audio = 0 because the track is silent." The model can learn to route around an absent modality instead of trusting a fake zero.

Fix part 2 — rehearse the hole during training: Modality Dropout. Randomly zero out a whole modality for a fraction of training examples (the multimodal analogue of ordinary dropout). The model is forced to make good predictions from any subset of modalities, so a missing modality at serve time is a state it has seen thousands of times, not a novel shock. This is also what makes the gate's "skip a modality under load" trick safe — the skip lands the model in a rehearsed state. Modality Dropout is the single highest-leverage robustness trick in this lesson: it is cheap, it is one line in the data loader, and it converts a serve-time cliff into a graceful degradation.

Fix part 3 — fill the hole with a real proxy. When you want better than "route around it," synthesize a plausible vector for the absent modality. Per-modality recipes that work in production:

Modality conflict is a feature, not noise

The instinct when two modalities disagree is to average them away. That throws out signal. When the title screams one thing and the pixels show another, the disagreement itself is the most informative thing about the item — most often a clickbait tell. Treat conflict as an explicit input.

• Clickbait contradiction feature. Compute a cross-modal consistency score (the cosine of the title-text vector against the visual-content vector in the aligned space). A sharp divergence — sensational title, mundane or low-quality frames — fires a contradiction feature the ranker can use to discount the item, and an aligning loss can train the two towers to agree on honest items. This is exactly what the consistency score in the "Where it bites" box is for, promoted from a footnote to a named feature.
• OCR-vs-ASR disambiguation. ASR mishears; on-screen text is ground truth. Run OCR on the frame to correct ASR errors — when the two text channels disagree, OCR (what's literally printed) usually wins, and the corrected transcript feeds a cleaner text vector. The disagreement is a repair signal, not a problem to suppress.
• Multimodal joint moderation. A single modality over-flags: a banned word in the title but a benign frame is probably a false positive (a cooking clip titled with a slang term). Joint detection — keep it if the visual is harmless — cuts false bans. The conflict between the text flag and the visual content is what lets you make the right call.

Cold-start hybrids beyond CLIP

A CLIP content vector is the first lifeline for a cold item, but it is not the only architecture, and a senior answer names the complements. None of these re-derive the cold-start problem — see 15 · Cold start and 38 · Cold start (advanced) for the full treatment; here is how the multimodal stack plugs into them.

• GNN content-graph propagation (GAT). Insert the new item into the user–item interaction graph as a node whose features are its multimodal content vectors, then propagate with a graph attention network. A long-tail item with zero clicks inherits the embeddings of its high-quality, well-understood neighbors — content similarity becomes a bridge across the interaction graph, so the cold item borrows warm items' signal through the edges rather than starting isolated.
• KNN pseudo-labels. When the target item's behavior is too sparse to train on, borrow the historical interactions of its content-nearest neighbors as pseudo-labels — a warm start for the cold item's behavioral features.
• MAML fast-adapt. Meta-learn an initialization that adapts to a new item or user in ~5 interactions rather than the hundreds a cold ID embedding needs. The model learns how to learn a new item fast, so the first handful of clicks move it a long way.
• Adversarial popularity-disentangling. In the item tower, adversarially strip the popularity signal so a fresh, low-exposure item isn't crushed by a "this is unpopular" feature before it has had a chance — it competes on content, not on a head start it can't have.

The engineering bill — encoding is the expensive part

The catch: running a ViT over every frame of every uploaded video is orders of magnitude more expensive than a table lookup. The discipline that makes multimodal practical is precompute offline, serve a vector.

• Encode once, at upload. The instant a clip is ingested, an offline/nearline pipeline runs the encoders and writes its content vectors to a feature store. Online serving never touches a CNN — it reads a 256-dim vector, the same cost as any other feature.
• Keep d small and cached. 256-dim is a deliberate budget: at 10⁹ items, every extra dimension is real storage and ANN-index memory. Tiered caches (hot items in memory/Redis, the long tail recomputed or on disk) keep the footprint bounded.
• Distill heavy backbones. A TimeSformer is accurate but slow; distill it into a lighter student, or sample only key frames, so the upload-time pipeline keeps up with ingest QPS.
• Refresh on a schedule. Content vectors are quasi-static (the pixels don't change), so they update daily/weekly — unlike real-time behavioral features. The two feed the same ranker on different clocks.

The "behavioral features update in real time" clock is itself an engineering system, not a single arrow. Content vectors are encoded once and cached; the other clock — real-time CTR, dwell, scroll speed — runs on a streaming pipeline that a serving lesson like 19 · Large-scale optimization covers in depth, but the multimodal-specific shape is worth pinning down:

The pieces that matter for an interview: a dual-cadence update (real-time CTR refreshed ~every 10s via Flink State; slower interest clusters hourly); a tiered store (Redis for second-level features, HBase for minute-level); an atomic version-snapshot swap so inference always reads a consistent feature set, never a half-updated one; SLA tiers (core behavioral features must arrive in <500ms, auxiliary in <5s); a degrade-on-backlog rule (when the Kafka lag exceeds a threshold, fall back to the last hour's features rather than block); and a P99 serving target under 100ms, held with FP16/INT8 quantization, dynamic batching, and early-exit. Feature-warmup preloads hot-video features to the edge, and a KL-divergence monitor watches for feature drift.

Interview prompts you should be ready for