Data quality, EDA, and anti-fraud - trust the log before the model

The book gives data cleaning and EDA the same seriousness as model design. This lesson linearizes that operational layer: validate raw events, handle missingness and outliers, manage class imbalance, automate dirty-data detection, use EDA to understand long tails and segments, detect brush traffic, and decide when bad data blocks training or launch.

1 · Data quality is model quality

A recommender is trained on logs produced by product code, clients, networks, old rankers, creators, and sometimes attackers. When a metric moves, the first question is not always "what did the model learn?" It may be "what changed in the data?"

The book's data-cleaning sections emphasize missing values, abnormal values, duplicate data, non-structured noise, and business-specific invalid events. In recommendation, those are not just ETL nuisances; they directly define labels and features.

2 · Validate every boundary

3 · Treat missing values as information first

The book distinguishes deletion, filling, prediction, and special-event marking. In recommender logs, missingness often has meaning.

Always add a missingness indicator for important fields. A filled value without a missing flag hides the fact that the model is seeing uncertainty.

4 · Separate outliers, fraud, and valid long-tail behavior

The book lists 3-sigma, IQR, KNN, DBSCAN, isolation forest, and business rules. Use statistics to find candidates, then business logic to classify them.

5 · Handle class imbalance without breaking calibration

The book highlights severe positive/negative imbalance in click data. Most impressions are non-clicks; most users do not comment; conversions may be rare and delayed. If you sample carelessly, the model may rank better offline but produce miscalibrated scores online.

Keep validation and test sets close to real traffic distribution unless you have a specific counterfactual evaluation design. Sampling may help learning; it should not make evaluation fictional.

5.1 · Constructing negative samples is a data-quality decision

Before imbalance is a loss-function problem (focal loss, class weights - see 06 for the training-side sampling), it is a data-construction problem. In a feed, every impression that was not clicked is a candidate negative - but many of those impressions are garbage, and feeding garbage negatives teaches the model false negatives. The data layer decides which impressions are even allowed to become negatives, and how heavily each counts.

Two cautions carry over from above. Keep auxiliary labels (watch-time) attached while you sample on click, so CTR-driven undersampling does not silently wreck the watch-time objective. And because undersampling lifts the positive rate the model sees, online scores come back inflated - a calibration layer (cross-link 06) maps them back to true-distribution probabilities. The hard-negative keep-rule here is the data-side counterpart to in-batch hard-negative mining in 06; this lesson decides which logged rows qualify, 06 decides how to sample them in the loss.

6 · Choose the preprocessing engine by latency and state

The book asks candidates to compare distributed frameworks for TB-scale preprocessing. The answer should be practical, not tribal.

Mini-scenario: if the task is "clean last quarter of logs and rebuild training data," Spark is the natural answer. If the task is "detect watch_time > duration in the live stream and quarantine bad events before they reach features," Flink is the natural answer. Hadoop is usually the cold-storage fallback, not the frontline tool.

7 · Automate dirty-data detection in the stream

The book proposes an automated real-time dirty-data system with Kafka, schema validation, Flink/Spark Streaming, rule checks, model checks, repair/drop/degrade decisions, and monitoring. The linear architecture is:

7.1 · Model-based anomaly checks beyond rules

Static rules catch what you already know to forbid. They miss the anomaly nobody wrote a rule for - a feature that drifts 20% over a week, a new device tier whose watch-time distribution is subtly bimodal. Run a thin layer of statistical/model checks beside the rule engine, per numeric feature, cheap enough for the stream:

7.2 · Graded decision tiers, not one verdict

A binary accept/reject throws away good rows on a minor blemish and quietly admits fraud that "passed schema." Grade the response to the severity:

The flag travels with the row. A repaired value carries a repaired marker and a missingness indicator (section 3) so downstream models and audits know it was synthesized, not observed.

7.3 · Dynamic thresholds - why static bands false-alarm

This is the failure mode that pages on-call at 9pm on a holiday. A static rule like "alert if 1-minute event volume drops below X" is calibrated to an ordinary Tuesday. On Chinese New Year, Black Friday, or during a live event, traffic legitimately triples or craters - and the static band fires a flood of false alarms exactly when humans are least available. After a few false pages, on-call mutes the alert, and then the real incident slips through. Static thresholds train people to ignore them.

Mechanism: Prophet (or STL above) fits trend + weekly/daily seasonality + an explicit holiday term, and emits a prediction interval per timestamp. The "threshold" becomes a moving function of time instead of a constant. Same idea applied to volume, null-rate, and CTR turns "3-sigma off a global mean" into "3-sigma off the seasonally-expected mean."

7.4 · Layered monitoring - three gates, three owners

One dashboard of 200 metrics gets ignored. Split monitoring into layers that map to where damage happens, each with its own threshold and its own auto-response:

The split matters because the response differs by layer. A raw null spike is a logging incident; a feature-layer PSI alarm is a retraining trigger; a serving-layer mismatch is a launch blocker. Routing all three to one channel loses that signal. Feature importance should rank which features are "core" - monitoring all of them equally wastes compute, so prioritize the top features that actually move the model.

8 · Use EDA to understand the product story

EDA is not just charts. It asks whether the data matches the product mechanism. The book's EDA section focuses on long-tail watch time, brush traffic, missing values/outliers, group differences, and viewing-experience metrics.

9 · Detect brush traffic as biased training signal

Brush traffic is not only a security problem. It becomes fake preference data if it reaches training. The book's EDA answer suggests operation frequency, time patterns, device/IP aggregation, sequence anomalies, heatmaps, and anomaly models such as isolation forest or LOF.

9.1 · Put numbers on the thresholds

The table above is qualitative; an interviewer wants the actual flags. The EDA procedure is: compute a per-capita distribution, set a percentile cut, then escalate the survivors to an unsupervised model. Concrete defaults teams start from:

9.2 · Visualize, then score, then escalate

A time-density heatmap (time-of-day on one axis, user/account on the other) makes unnatural clustering pop out - a block of accounts all active at 3am, or all firing in the same 10-second column, that no organic audience produces. Use it to find candidate clusters; use a model to score them.

For scoring, run isolation forest or LOF (Local Outlier Factor) over the behavior vector (ops/min, watch-vs-like ratio, IP/device sharing, inter-event timing) and combine the unsupervised score with the business rules above - a row needs both a high anomaly score and a rule hit before it is quarantined, which keeps the false-positive rate down. The precision/recall trade-off (kill a real power user vs miss a bot) is validated by A/B, not assumed.

10 · Define incident severity for data problems

Interview prompts you should be ready for