Data engineering for recommendation

The model gets the credit. The pipeline does the work — and when it silently breaks, it kills the model with no error: stale features, train–serve skew, a null column that used to be populated, a date partition that never landed. A recommender is only ever as good as the data plumbing beneath it. This lesson is the offline/batch + data-quality substrate that lesson 17's streaming plane quietly assumes.

Why this is the unglamorous half that decides everything

A ranking model is a pure function: same features in, same score out. So when last week's AUC was 0.78 and today's is 0.71 with no code change, the model didn't rot — the data feeding it did. The failure mode that should keep you up at night is not the model throwing an exception; it is the pipeline that keeps running and keeps lying: a feature whose upstream join started returning NULL, an aggregate computed over a partition that arrived three hours late, a column whose distribution drifted because a logging SDK shipped a new version. The model dutifully consumes the garbage and produces confident, wrong scores. No alarm fires, because nothing crashed.

The batch big-data stack — MapReduce, Hadoop, Spark

Recsys data is too big for one machine: years of impressions, billions of events a day. The whole field is built on one primitive — MapReduce — and one realization about why it was slow.

MapReduce: map → shuffle → reduce

Express a computation as two functions over key-value pairs. Map runs in parallel on each input split, emitting (key, value) pairs. Reduce runs once per key, aggregating all values for that key. Between them sits the part nobody advertises and everybody pays for: the shuffle, which sorts and moves every mapper's output across the network so that all values for a given key land on the same reducer.

Map and reduce are embarrassingly parallel and run on local data. The shuffle is the bottleneck because it is the only all-to-all step: it serializes intermediate output to disk, sorts it, and ships it across the network. Roughly, if mappers emit N records that must be regrouped onto R reducers, the shuffle moves O(N) bytes over the network and does O(N log N) sort work — and it cannot start reducing until enough of it has landed. Minimizing shuffle volume is the single highest-leverage batch optimization (combiners that pre-aggregate map-side, broadcast joins that avoid shuffling the big side, columnar pruning so you shuffle fewer columns).

Hadoop/HDFS/YARN vs Spark — why Spark won iterative ML

Classic Hadoop MapReduce writes every intermediate result to HDFS (the distributed disk) between stages, with YARN scheduling containers. That is fine for a single pass but catastrophic for ML: training, feature aggregation, and graph algorithms are iterative, and Hadoop re-reads the dataset from disk on every pass. Spark's core insight is to keep the working set in memory across iterations.

Three Spark concepts earn the speedup. The RDD/DataFrame is the in-memory partitioned dataset. The lazy DAG means transformations (map, filter, join) build a plan but execute nothing until an action (count, write) forces it — so the optimizer can fuse stages and prune columns before any work runs. Lineage is the recovery story: instead of replicating data, Spark remembers the DAG that produced each partition, so a lost partition is recomputed from its parents rather than re-read from disk. Cache the training matrix once with persist() and every gradient pass reads RAM — that is the 10–100× over Hadoop on iterative work.

Data skew and the straggler problem

Recsys data follows a power law: a handful of items and users dominate. Partition by item_id and the viral clip's key sends a colossal value list to a single reducer while the others idle. Batch jobs run as fast as their slowest task, so one hot key makes wall-clock time blow up even though average load is tiny.

T_job = max_r load(r) ≫ mean_r load(r) when one key is hot

The standard fix is salting: append a random suffix to the hot key (itemA#0 … itemA#k) so its traffic spreads across k reducers, then do a second small reduce to merge the k partials. You trade an extra cheap aggregation for a flat load profile. Other levers: tune partition count (more partitions → finer granularity), isolate known hot keys into their own path, prefer map-side/broadcast joins so the skewed side is never shuffled. The widget below lets you feel the straggler problem and watch salting flatten the max.

The named tuning levers — and the failure mode each one fixes

Everything above is conceptual: shuffle is expensive, skew makes stragglers, salt the hot keys. In an interview the follow-up is always "okay, what do you actually turn?" Each Spark/Hadoop knob below exists to attack one of the failure modes already on the table — shuffle volume, skew, or the small-files problem. Anchor it to a real win: a feature build over 2 TB of daily user-behavior logs (impression × user-stats join) went from 4 hours to 35 minutes after four changes — none of them touched the algorithm.

Stacking these on the 2 TB job: a broadcast join of the small user-stats dimension table eliminated the big side's shuffle entirely; spark.sql.shuffle.partitions=2000 right-sized the remaining shuffle so nothing spilled; persist(MEMORY_AND_DISK) kept the cleaned input resident across the multi-stage feature derivation instead of recomputing it; and repartition(200) on write collapsed thousands of tiny part-files into a clean output. Net: 4 h → 35 min (≈ 7×), no model change. On the Hadoop side the same instincts have older names — mapreduce.task.io.sort.mb and mapreduce.reduce.shuffle.parallelcopies tune the shuffle buffers, speculative execution relaunches a straggler's task on a second node, and CombineFileInputFormat is the small-files fix at read time. (Skew itself is fixed with salting — derived in the section above, not repeated here; broadcast and bloom pre-filter are the levers that let you avoid shuffling the skewed side in the first place. For the streaming-plane equivalents of these knobs — partition sizing, backpressure, state backends — see lesson 36; for where these computed features physically land, see the columnar/partitioning section below and the feature deep-dive in lesson 35.)

Lambda vs Kappa — and the train–serve skew it causes

Because most pipelines need both throughput and freshness, the classic architecture runs two layers in parallel. This is Lambda, and its convenience hides the single most common silent killer in production recsys.

In Lambda the same feature — say "user's 7-day average watch time" — is implemented twice: once in the batch layer's SQL, once in the speed layer's stream code. The two implementations inevitably diverge: a different rounding rule, a different window boundary, a different null handling. The model trains on the batch version and serves on the speed version, and the gap between them is train–serve skew — the model sees feature distributions at serving time that it never saw in training, and quality degrades with no error anywhere. (Same disease diagnosed from the serving side in lesson 18 and the feature-definition side in lesson 28.)

Kappa is the structural answer: there is no batch layer. Everything is a stream over the append-only log; a "batch recompute" is just replaying the log from offset zero, run by the same code that serves live traffic. One definition, one codepath, no skew by construction — the Kappa argument from lesson 17. The cost is that all logic must fit the streaming model and replays of years of history can be slow, so many shops keep a thin batch layer for cold backfills and accept the discipline of generating both from one shared transformation library. The principle stands: one feature, one definition.

ETL and the feature pipeline — Extract, Transform, Load

The plumbing that turns raw logs into model-ready features is an ETL (or ELT) pipeline. Three stages:

• Extract — pull from sources: the Kafka event log, application DBs (CDC), third-party feeds, the existing warehouse. The job here is faithful capture with provenance, not cleverness.
• Transform — the bulk of the work: parse, dedup, join behavioral events to entity attributes, compute aggregates and windows, encode features (see lesson 28), attach labels. This is where most bugs and most skew are born.
• Load — write to the destination: the offline feature store / warehouse tables for training, and the online KV store for serving. ELT flips the order — load raw, transform in-warehouse — which is the modern lakehouse pattern when storage is cheap and compute is elastic.

Warehouse, lakehouse, and the storage layout that decides your scan cost

Where the data lands matters as much as how it is computed. A data warehouse (BigQuery, Snowflake) stores cleaned, schema-on-write tables; a data lake stores raw files cheaply (schema-on-read); a lakehouse (Delta/Iceberg/Hudi) puts warehouse-grade transactions, schema evolution, and time-travel on top of lake-cheap object storage — the dominant recsys pattern because it serves both the training backfill and the analyst's query from one copy.

Two layout choices dominate query cost:

• Columnar formats (Parquet / ORC). Behavioral tables are wide but queries touch few columns. Columnar storage keeps each column contiguous, so reading watch_time and label from a 200-column table touches ~1% of the bytes. It also compresses far better (a column is homogeneous — run-length / dictionary encoding on low-cardinality fields like country), so less I/O and less network on the shuffle.
• Partitioning (Hive-style, usually by date). Physically lay the table out as /dt=2026-05-30/… directories. A query for "last 7 days" reads 7 directories and the engine prunes the rest without opening a single file — the scan cost drops from "all history" to "the days you asked for." Date is the canonical partition key in recsys because almost every training query and every aggregate is time-bounded. Over-partitioning (e.g. by user_id) creates millions of tiny files and destroys throughput — the small-files problem — so partition coarse, sort fine.

The offline feature store and point-in-time-correct joins

The offline feature store is the warehouse side of the contract whose online side is lesson 17's KV store. Its hardest job is building a training set without leaking the future. To train on an impression at time t, every feature value you attach must be the value as it was known at t — not the latest value. A naive join of the impression table to a "current user stats" table is a look-ahead leak: you train on a 7-day CTR that includes clicks that happened after the impression, get a gorgeous offline AUC, and watch it collapse in production because that future never exists at serving time.

The fix is a point-in-time (as-of) join: for each label row at t, join to the most recent feature snapshot with timestamp ≤ t. This is exactly the leakage discipline of lesson 28, enforced at the pipeline level: features carry event-time, and the training-set builder asks "what did we know then?", never "what do we know now?"

Data quality — proving the data is still good

The processing stack delivers data fast; data-quality monitoring is what stops it from delivering bad data fast. The mental model: every feature pipeline is a contract, and you assert the contract on every run. Garbage feature in → silent model rot. The four classes of assertion:

PSI — quantifying distribution drift

The standard recsys drift metric is the Population Stability Index. Bin a feature into B buckets; compare today's proportion a_i in each bucket to a reference (training, or last week) proportion e_i:

PSI = Σ_i=1..B (a_i − e_i) · ln(a_i / e_i)

It is a symmetrized KL-style divergence. Rule of thumb: PSI < 0.1 stable, 0.1–0.25 moderate drift (investigate), > 0.25 significant drift (the model's training distribution no longer matches reality — retrain or alert). Compute PSI per important feature on every batch and you catch the "logging SDK changed and a feature's distribution shifted" failure before AUC ever moves. This is the offline twin of the freshness SLA: freshness watches when data arrives, PSI watches what it looks like when it does.

Kafka consumer-lag / backlog — the streaming-side smell

On the streaming plane (defer to lesson 17 for internals), the headline data-quality signal is consumer lag: the gap between the latest offset written to a Kafka topic and the offset your stream processor has consumed. Rising lag means the processor cannot keep up — features are getting staler by the second even though nothing has crashed. Root-cause it in order: a traffic spike (transient), a slow downstream sink (the store can't absorb writes), a skewed partition (one hot key — same straggler problem, now in streaming), or an under-provisioned job. The protective mechanism is backpressure: the processor signals upstream to slow down rather than OOM, and the durable log buffers the backlog so you degrade instead of dropping events. Alert on lag, not just on crashes — a job that is alive but 20 minutes behind is serving stale features silently.

EDA and cleaning as a skill

Before any of this is automated, someone has to look at the data. Exploratory data analysis is not a junior chore — it is where you find the bot traffic, the broken column, and the long tail that will wreck a naive average. The recurring recsys diagnoses:

• Long-tail / heavy-tail. Watch-time, follower counts, item popularity are all power-law. A linear-axis histogram looks like a single spike at zero with nothing else visible. Always plot counts on a log scale (or log-log) — the tail becomes a readable line, and you can decide whether to log-transform the feature (lesson 28) so the model isn't dominated by a few whales.
• Missing values — strategy depends on why it is missing (table below). The senior point: missingness is often itself a signal (a user who never set a profile field behaves differently), so add an explicit is_missing indicator rather than silently imputing it away.
• Outliers — a 14-hour "watch time" is a bug or a bot, not a superfan. Detect with IQR (outside Q1−1.5·IQR … Q3+1.5·IQR), z-score (|z| > 3, but only valid if roughly normal — useless on heavy tails), or isolation forest for multivariate anomalies. Then decide: cap (winsorize), drop, or keep-and-flag — capping is usually safest for skewed features.
• Bot / fraud signals — superhuman click rates, impossibly regular inter-event timing, one device hammering thousands of accounts, click-without-dwell. These poison both labels and aggregates; filter them in the cleaning stage or they teach the model to recommend whatever the bots inflate.

Class imbalance — the CTR ≈ 1% problem

CTR labels are extreme: roughly one positive per hundred impressions. A model can score 99% "accuracy" by predicting "no click" always — and be useless. The imbalance ratio is the headline:

imbalance ρ = N_neg / N_pos ≈ 99 : 1 (CTR ≈ 1%)

Three families of remedy, from blunt to preferred:

Interactive · data-skew & salting simulator

Distribute a fixed number of keys across N reducers under a tunable Zipf skew. Without salting, hot keys pile onto one reducer and job time = the slowest reducer's load. Toggle salting to split hot keys across reducers and watch the max flatten. The speedup is the ratio of the two job times — all computed live.

Interview prompts you should be ready for