Feature engineering as a discipline

"Deep models learn the crosses, so feature engineering is dead" is the most confidently-wrong thing a candidate can say. The embedding table learns interactions among features you give it. It cannot conjure a signal you never extracted, it cannot un-leak a target you smuggled in, and it cannot fix a high-cardinality column that blew your table to 40 GB. This lesson is the craft that lives below the model.

What the deep model actually automated — and what it didn't

Lesson 04 told the architecture story: LR → FM → Wide&Deep → DLRM → DCN-v2 is the history of moving feature crosses from hand-written recipes into learned cross-layers. That is real, and it killed the era of engineers typing query_token × ad_id by hand. But it automated exactly one thing — the interaction between features already present in the input. Everything upstream of that is untouched and still on you.

The mental model: the cross-network is a powerful interaction engine sitting on top of a representation you built. Give it garbage — a leaky target encoding, a min-maxed feature whose serving-time max is unknown, a 10⁹-cardinality column you dumped raw into an embedding table — and it will faithfully learn the interactions of your garbage. Feature engineering didn't move into the model. It moved into the plumbing: encoding, scaling, leakage control, and serving parity.

Categorical encoding — the cardinality decides everything

A categorical column has no native numeric meaning, so you must choose how it enters the model. The wrong choice either injects fake structure, explodes dimensionality, or leaks the label. The deciding variable is almost always cardinality — how many distinct values — and the secondary one is whether the model is a tree (splits on order) or something distance/dot-product based.

One-hot vs label/ordinal — the false-order trap

One-hot maps each of k categories to a basis vector — no spurious order, but k new columns. Fine at k=12 (device type), insane at k=10⁸ (user_id). Label / ordinal encoding maps categories to integers 0,1,2,…. That is correct only when the values have an order (size = S<M<L). For a nominal feature — country, ad_creative_id — it injects a lie: it tells a linear model or a distance metric that country=3 is "between" 2 and 4 and "twice" 1. Trees survive it (a split is just x < t, and the model can chop the integer line into the right buckets given enough depth), which is exactly why label encoding is a defensible default for GBDTs and a bug for everything else.

Frequency / count encoding

Replace each category with how often it appears: enc(c) = count(c) or its normalized frequency. Cheap, one column, no leakage of the label (it uses only the feature distribution). It encodes a real signal surprisingly often — a rare merchant and a giant marketplace behave differently — but it collides two categories with the same frequency into the same number, so use it as a supplement, not a replacement for identity.

Target / mean encoding — powerful, and a loaded gun

Replace each category with the mean label observed for it: for a CTR problem, enc(c) = E[y | c], the empirical click-through-rate of that category. This is wildly effective — it compresses a high-cardinality column into one column that is already half the prediction. It is also the single most common source of target leakage in applied ML, because the encoding of row i is computed from a statistic that includes row i's own label.

The leak is sharpest for rare categories. A category that appears once gets enc(c) = y_i — the encoding is the label. The model learns "when this feature equals 0.0, predict 0; when it equals 1.0, predict 1," scores a gorgeous training AUC, and collapses in production where that category's encoding is computed from other rows. Two fixes, and you need both:

• Out-of-fold (K-fold) encoding. Split the training data into K folds. To encode a row in fold j, compute the category means using only folds ≠ j. The row's own label never enters its own encoding. At serving time, encode with statistics from the entire training set. (This is the same hold-out logic that makes 08 · Evaluation's offline numbers trustworthy.)
• Smoothing toward the global mean. A category seen 3 times has a noisy mean; shrink it toward the global rate ȳ with a prior weight m:

enc(c) = ( n_c · ȳ_c + m · ȳ ) / ( n_c + m )

where n_c is the category's count, ȳ_c its raw mean, ȳ the global mean, and m a smoothing strength (effectively "how many imaginary global-average observations to add"). When n_c is large the encoding trusts the category; when it is tiny it falls back to the population rate — which is exactly the right Bayesian behavior, and structurally the same shrinkage you saw stabilize cold-start priors in 15 · Cold start.

The hashing trick — when cardinality is the enemy

Lesson 04 introduced this for the embedding table: a deterministic hash h(c) mod B maps an unbounded category space into B fixed buckets, so a 10⁹ user_id space fits in 10⁶ rows with no vocabulary to maintain and zero handling needed for unseen IDs at serving. The cost is collisions — two distinct IDs share a row and become indistinguishable to the model. You trade a controlled amount of identity for a bounded memory footprint; you size B so that popular values rarely collide (collisions in the cold tail are cheap, collisions among your top advertisers are not). Double-hashing or a per-feature hash seed reduces systematic collisions.

Hierarchical / taxonomy encoding

Interest tags are rarely flat — they live in a tree: sports → basketball → NBA → Lakers. A flat one-hot of the leaf wastes the structure and starves rare leaves of data. Encode the path: emit a (coarse, mid, fine) tuple — category, subcategory, tag — each its own embedding, and let the model pool them. A cold leaf tag still inherits a well-trained category embedding, so it degrades gracefully toward its parent rather than to noise. This is target-encoding's smoothing idea expressed structurally: borrow strength from the parent when the leaf is sparse.

That paragraph is the intuition; below are the four named schemes you should be able to reach for by name, and — more importantly — why flat encoding actively fails on a deep taxonomy in a way none of the schemes above (one-hot, target, hashing) repair on their own.

Four named hierarchical schemes

1 · Path encoding. Serialize the full path to a single token — "electronics/phone/android" — then hash or one-hot that string. It preserves the complete ancestry in one feature and is trivial to compute online (it is just a string concat + hash, far cheaper than a graph embedding). The catch is that it worsens sparsity rather than curing it: electronics/phone/android and electronics/phone/ios are as unrelated to the model as two random hashes — the shared electronics/phone prefix buys you nothing, because the model never sees the prefix as its own feature. Path encoding is best as a supplementary high-precision identity alongside the level features below, not a standalone fix for the tail.

2 · Parent-conditioned target encoding with per-level Bayesian smoothing. This is the scheme that directly repairs the long-tail failure above, and it is the hierarchical generalization of the out-of-fold + smoothed target encoding from the section above. Instead of smoothing every level toward the global mean, you smooth each level toward its parent's encoded mean, walking down the tree: target-encode L1 (electronics) toward the global rate; then encode L2 (phone) toward the L1 estimate; then the leaf (pixel-8-128gb) toward its L2 parent. Each level uses the same shrinkage you saw for flat smoothing, but with the parent mean in the prior slot:

enc(leaf) = ( n · leaf_mean + α · parent_mean ) / ( n + α )

where n is the leaf's own count, leaf_mean its raw observed CTR, parent_mean the (already-smoothed) encoding of its immediate parent, and α the per-level smoothing strength — the count at which the leaf is trusted half-and-half against its parent. The one change from flat smoothing is the prior: a rare leaf now falls back to phone's well-measured rate, not to the platform average — which is exactly the signal flat target encoding threw away. (Keep the same out-of-fold discipline at every level: each level's means are computed from folds that exclude the row being encoded, or the leak you fixed flatly re-opens hierarchically.)

3 · Graph embedding of the taxonomy tree (Node2Vec). Treat the taxonomy as a graph — nodes are categories at every level, edges are parent–child links — and run Node2Vec (the random-walk embedding from 24 · GNN) over it to learn a dense vector per node where tree-adjacent categories land near each other in space. A cold leaf's vector is then close to its siblings' and parent's by construction, so it shares strength geometrically rather than via an explicit shrinkage formula. This is the most expressive option and composes cleanly with behavior features (concatenate the node vector with the user/item embeddings), but it is the heaviest to maintain: re-running Node2Vec when the taxonomy changes is an offline batch job, which is why path/level encodings are preferred when the online compute budget is tight.

4 · Multi-granularity bucketing. Encode different levels with different schemes matched to their cardinality: L1 (~20 top categories) as plain one-hot, L2 (~500) and the leaf as learned embeddings, and feed all of them. The model pools a cheap exact L1 signal with richer learned L2/leaf vectors. The one caveat the source flags: keep the dimensions consistent where you intend to add or attention-pool across levels — if L1 is a 20-d one-hot and the leaf is a 32-d embedding you cannot sum them, so either project to a common width or keep them as separate concatenated fields the model reconciles itself.

Cold leaves: parent → child transfer init

The four schemes handle a leaf that has some data; a brand-new leaf (a SKU added this morning, a freshly-minted hashtag) has none, and its embedding row is still at random init. Don't ship random noise — initialize the new leaf's embedding from its parent's (copy phone's vector into the new pixel-9 row, optionally with a little jitter), so from its very first impression it predicts as a generic phone and then specializes as data arrives. This is the embedding-space twin of the n=0 case of the shrinkage formula — at n=0, enc = parent_mean exactly — and it is the same cold-start strength-borrowing you saw in 15 · Cold start, here applied structurally through the taxonomy instead of through a content model. (The deeper deep-dive on entity-organized feature stores and these encoders lives in 35 · Feature engineering deep dive.)

When to embed vs encode

The senior heuristic: embed if you have a deep model and enough exposures per value to train a vector; target-encode (OOF + smoothed) if you are on a GBDT or the cardinality is high but exposures are thin; hash when the vocabulary is unbounded or memory-bound. These compose — it is normal to hash then embed, and to feed both an embedding and a smoothed target encoding of the same column.

Numerical features — scaling, skew, bins, and the streaming wrinkle

Scaling, and why it matters for some models and not others

Three standard scalers. Standardization (z-score) centers and unit-variances a feature:

z = ( x − μ ) / σ

Min-max squashes to [0,1] via (x − min)/(max − min); robust scaling uses the median and IQR, (x − median)/IQR, so a few wild outliers don't dominate. The crucial fact interviewers probe: scaling matters enormously for some models and not at all for others.

• Linear models, neural nets, distance/kernel methods (kNN, SVM, k-means): scaling is essential. Gradient descent on un-scaled features takes a ravine-shaped loss surface and zig-zags; a regularizer like L2 penalizes large-scale features more than small-scale ones purely because of their units; a Euclidean distance is dominated by whatever feature happens to be measured in big numbers. Scale, or the optimizer and the metric both lie.
• Tree models (GBDT, random forest): scaling is a no-op. A split is x < t; it is invariant to any monotone transform of x, so z-scoring, min-maxing, or doing nothing all produce identical trees. This is why XGBoost users skip scaling entirely — and why a candidate who says "always standardize your features" reveals they don't know which model they're feeding.

Skew handling — log, Box-Cox, Yeo-Johnson

Prices, counts, watch-times, and bids are heavy-right-tailed: a few huge values, a long thin tail. A linear or NN model handles a roughly-symmetric input far better, so you compress the tail. The workhorse is log1p, log(1+x), which handles zeros (where plain log x blows up) and turns multiplicative structure additive. For a principled, data-fit transform there is the Box-Cox family,

(x^λ − 1) / λ for λ ≠ 0, and ln x for λ = 0

which finds the power λ that best normalizes the data — but Box-Cox requires x > 0. When the feature has zeros or negatives (a net P&L, a signed delta), reach for Yeo-Johnson, the generalization that is defined on the whole real line. The discipline: pick the transform on the training distribution, freeze λ (and μ, σ), and apply the frozen parameters at serving — re-fitting at serving time is itself a leak.

Binning / discretization — and why bucketizing a monotone feature helps

Discretization replaces a continuous value with the bucket it falls in. Equal-width bins cut the range into uniform intervals (sensitive to outliers — one huge value stretches every bin). Equal-frequency / quantile bins put the same count in each bucket (robust, the usual choice). Lesson 04 already flagged the punchline: log(1+price) in 32 quantile buckets often beats raw price, because a model that can't bend a feature can be handed the bend pre-made. An LR or a shallow MLP fits a monotone-but-curved CTR-vs-age relationship slowly; bucketize age and give each bucket its own weight/embedding and the model expresses the curve trivially. Trees don't need this (they bin internally), but for everything linear it converts a hard non-linear fit into a free one — at the cost of within-bucket resolution.

Online / streaming normalization — you don't have a global mean

Every formula above assumes you can compute μ and σ over the whole dataset. In a streaming system (17 · Real-time streaming) you cannot — the data arrives forever and the "global mean" doesn't exist yet. Two consequences:

• Use running statistics. Maintain μ, σ² with Welford's online update, or an exponentially-weighted moving average so recent data dominates. For quantile bins you cannot store every value, so you approximate the distribution with a sketch (t-digest, KLL) and read bucket edges off it.
• Watch for distribution drift. A z-score computed from last month's μ, σ silently mis-scales today's data when the distribution shifts (a sale, a new market, a logging change). The running statistic adapts — but now your training stats and your serving stats are computed by different code paths over different windows, which is precisely the train-serve skew the next section is about. The safest pattern: the same streaming normalizer object writes the feature at training-log time and at serving time, so there is one code path and the skew cannot open up.

Feature selection & importance — three families and the right default

More features is not more signal. Noise features add variance, inflate serving cost, widen the leakage surface, and (for linear models) wreck coefficients via collinearity. Selection methods fall into three families:

Permutation importance is the right model-agnostic default. Train the model, measure validation score, then shuffle one feature's column and measure how much the score drops — a big drop means the model relied on that feature. It works on any model, measures importance as actually used, and needs no retraining. Its sharp caveat: under correlated features it under-credits both. Shuffle feature A while its near-duplicate B stays intact and the model leans on B, so A looks unimportant — and vice versa — so two genuinely-important correlated features can both score near zero. The fix is to permute correlated groups together, or to reach for SHAP (21 · Explainability), which attributes a prediction across features with better-behaved theory and is increasingly the selection-and-debugging tool of choice.

Multicollinearity and VIF — a linear-model disease

When features are linearly redundant, a linear model's coefficients become unstable: the fit can put a giant positive weight on one and a giant negative on its correlate, and they nearly cancel — the prediction is fine but every coefficient is nonsense, so any story you tell from the weights is fiction. The diagnostic is the Variance Inflation Factor: regress feature j on all the others, take its R²ⱼ, and

VIFⱼ = 1 / ( 1 − R²ⱼ )

A VIFⱼ near 1 means feature j is independent of the rest; > 5–10 flags serious collinearity. The senior nuance interviewers love: collinearity wrecks linear-model coefficients, not tree predictions. A GBDT just picks one of the redundant features at each split and predicts identically — the prediction is robust; it's the interpretation (and any linear coefficient) that breaks. So VIF matters when you're shipping or explaining a linear model, and is largely irrelevant to a deep ranker's accuracy (though redundant features still cost you serving latency and memory).

Validating a new feature — the only test that counts

A feature earns its place in two stages. Offline ablation: train with and without it on the same data and compare a held-out metric (AUC, log-loss). If it doesn't move the offline number, it almost never survives. But offline lift is necessary, not sufficient — it can come from leakage. Online single-feature A/B: ship the model with the one feature added against the same model without it, and read the real-traffic lift (08 · Evaluation). A feature that wins offline but is flat or negative online is the classic signature of a feature that leaked in training or isn't computable the same way at serving. The two-stage gate is non-negotiable for any feature you intend to keep.

The cross-cutting sin: leakage

Every section above circled the same drain. Leakage is information in your training features that will not be available, in the same form, at the moment you must predict — so your offline metric is measuring a fantasy. It wears three faces:

• Target leakage. A feature is a proxy for the label itself. Naive target encoding (a row's encoding contains its own y) is the canonical case; so is "used a promo code" predicting "made a purchase" when the code is only issued after purchase. Tell: an implausibly high single-feature AUC.
• Look-ahead / time leakage. A feature aggregates data from after the prediction timestamp. Computing a user's "30-day click count" over a window that extends past the event, or building a session feature that peeks at clicks later in the session, leaks the future. The fix is a strict point-in-time join: every feature value must be as-of the event time, never later — the same discipline 13 · User behavior sequences enforces so a sequence model never attends to events that hadn't happened yet.
• Train-serve skew. The feature is computed by one code path in the offline pipeline and a different one online — different default for a missing value, a stale window, a units mismatch, a min-max scaler whose serving max differs from training. The model trained on one distribution scores another.

Interactive · the target-encoding leakage demo

A high-cardinality categorical (think 200 merchant IDs, many seen only a handful of times) target-encoded against a binary label. Toggle naive vs out-of-fold (K-fold) encoding and slide the smoothing strength m. Watch the train-vs-validation AUC gap: naive encoding posts a gorgeous training AUC and collapses on validation — it memorized the label through the encoding. OOF + smoothing closes the gap to the feature's real signal. The numbers are computed live from a simulated dataset.

Interview prompts you should be ready for