Data, labels, and features — the material flow

If 03c was the map, this is the supply chain. General ML systems fail when the model trains on facts that were not available at decision time, serves on features computed differently than training, or optimizes labels that arrive late and biased. This lesson makes the data contract explicit: entity, time, label, feature, freshness, and parity.

The invariant

For every training example, the feature vector must be exactly what the online system would have known at that example's decision time. That is point-in-time correctness. Break it and you leak the future; your offline score becomes a fantasy.

Think like a timeline

Imagine a transaction at 12:00. At 12:00 the system may know the card, merchant, amount, device, and recent activity. At 12:05 it may learn the user complained. At day 23 it may learn the chargeback label. Training is only honest if the row for the 12:00 decision uses facts knowable by 12:00. Everything learned later is a label or feedback signal, not a serving-time feature.

0 · The data-contract recipe

For every ML design, write one row in this shape before talking about model architecture:

one example = entity + decision_time + features_known_then + label_learned_later

Then ask the questions in order: What is the entity? Which clock defines the decision? Which feature windows end before that clock? When does the label mature? How fresh must each feature be online? If you cannot answer those, the model may look good offline and fail in production.

1 · Name the entity and the clock

Every feature and label needs a grain: "one row per what?" The grain is usually an entity at a decision time: user at request time, transaction at authorization time, item at ranking time, store-SKU at forecast day. Without this, features become impossible to join correctly.

Concept	Question	Example
Entity key	What object is the prediction about?	user_id, txn_id, item_id, store_sku
Decision time	When did the system need the prediction?	Checkout authorization timestamp
Event time	When did source facts happen?	User clicked item at 12:03:14
Ingestion time	When did the system learn the fact?	Click arrived in Kafka at 12:03:22
Label time	When did truth become known?	Chargeback filed 23 days later

The hardest bugs hide between event time and ingestion time. If a feature says "transactions in the last hour" but includes an event that arrived five minutes after the decision, offline training has seen the future. In fraud, forecasting, ads, and recommendations, this can move metrics enough to ship a broken system.

2 · Feature stores are parity tools, not magic databases

A feature store exists to keep two copies of the same feature definition aligned: an offline store for training/backfills and an online store for low-latency serving. If the two stores are fed by different logic, you have created training-serving skew with nicer branding.

Same feature, two copies

Suppose the feature is "user clicks in the last 10 minutes." The offline copy is used to build historical training rows. The online copy is used at request time in milliseconds. They must share the same definition: which clicks count, which clock cuts the 10-minute window, what to do with late events, and what default to use if the value is missing. The store is not the magic; the shared definition is.

Feature class	Serving requirement	Failure mode
Static profile	Minutes to hours freshness	Stale segment or deleted user data retained
Aggregates	Windowed counts with event-time watermarks	Late events double-counted or missing
Real-time session	Milliseconds to seconds freshness	Online feature unavailable at p99
Embeddings	Versioned with model and index	Embedding model changes but ANN index does not
LLM/RAG context	ACL-aware retrieval and cache invalidation	User sees stale or unauthorized document

3 · Labels are delayed, biased, and product-shaped

Labels are not truth falling from the sky. They are produced by the product and by human or system processes. Design them as carefully as features.

Label problem	Example	Design response
Delay	Fraud chargeback after 30 days	Train with mature labels; use proxy labels for early warning, not final truth
Selection bias	Only shown items can be clicked	Exploration traffic; IPS/doubly robust estimates (ways to reweight biased logs); counterfactual logs
Intervention bias	Blocked fraud has no chargeback label	Random audit sample, manual review, reject inference (estimating labels for cases you blocked)
Human noise	Moderation labels disagree	Adjudication (tie-break review), gold tasks (known-answer checks), rater calibration, label confidence
Proxy mismatch	Click improves but retention falls	Multi-objective eval and online guardrails

The label-delay wall

A fraud model with 30-day mature labels cannot honestly retrain every hour on final labels. You can update features hourly, score online in milliseconds, and monitor proxies in real time, but the supervised target arrives on a different clock. The model lifecycle must respect that clock or it trains on biased partial truth.

4 · Freshness is an SLO

For each feature, specify a freshness SLO the same way 03 specified TTFT and TPOT. The right number comes from product sensitivity, not engineering taste:

feature_age_at_decision = decision_time - max_event_time_in_feature

Choosing freshness from first principles

Ask how quickly a stale fact can change the action. If fraud velocity changes in seconds, a 10-minute-old count can approve a bad transaction; freshness must be seconds. If churn outreach happens once a day, an hour-old feature rarely changes the action; batch is fine. Numeric example: a feed request at 12:00 using session clicks only through 11:59:45 has 15 s feature age. If the SLO is "session intent fresher than 30 s," it passes; if the SLO is "fraud velocity fresher than 5 s," it fails.

Use case	Feature freshness SLO	Why
Card fraud velocity count	Seconds	A burst of attempts is the signal
News feed interest	Seconds-minutes	Session intent changes while user scrolls
Ad budget pacing	Seconds-minutes	Overspend is irreversible
Churn prediction	Hours-days	Decision cadence is slow
Inventory forecast	Daily batch often enough	Action horizon is days or weeks

Freshness also sets architecture. Seconds usually means stream processing and an online store. Days can be batch ETL. If a feature has a one-day action horizon, building a sub-second streaming path is cost without product value.

5 · The leakage checklist

Before trusting an offline metric, run this checklist. If any answer is "not sure," the metric is not a release signal.

Availability: was every feature available before the decision time?
Windowing: are aggregates cut by event time with a late-event policy?
Identity: do train and serve use the same entity keys and joins?
Definition: is the same feature code or declarative definition used offline and online?
Backfill: can historical features be recomputed exactly after a code change?
Versioning: are feature definitions, embeddings, labels, and model artifacts linked?

What carries forward

Entity + time is the grain of ML design. Without it, joins leak, labels drift, and features cannot be reproduced.
Feature stores solve parity, not modeling. The definition must be shared; offline and online stores are materializations.
Labels have clocks and bias. Delayed, selected, intervention-shaped labels often bind before model capacity.
Freshness is an SLO. Put max feature age beside latency and cost in the requirements.