Applied & Classical ML · entry 02/06 · 3 min read

Features & leakage

Feature engineering is where most applied accuracy comes from, and leakage is where most impossible results come from — usually the same afternoon's work.

Features are the model's vocabulary

A model can only express relationships in terms of what it is given. Raw columns rarely say the useful thing: "account opened 2019-04-02" is inert, "days since opening" is predictive; twelve months of amounts are noise, "spend this month over median spend" is a signal. Applied machine learning is largely the craft of writing down what a domain expert already knows in a form a model can use — ratios, differences, counts over windows, flags for known-meaningful thresholds.

Leakage, defined precisely

Leakage is any information in your training features that would not be available, with those values, at the moment the prediction has to be made — or any way the outcome influenced which rows you have at all. It is not a subtle statistical concern; it is a time-travel bug, and it produces validation scores that feel like a breakthrough. The recurring shapes:

target leakage   a column computed after the outcome
                 ("num_collections_calls" for default prediction)
train/test leak  scaling or imputing before the split, so test
                 statistics inform training
group leak       the same customer in both train and test
temporal leak    random splits on time-ordered data, so the model
                 trains on the future and is tested on the past
duplicate leak   near-identical rows straddling the split

How to catch it

Suspicion is the main instrument, and the trigger is a score far above what the domain's published baselines achieve — an absolute threshold is no use, since on a rare-event problem a 0.99 AUC may mean nothing. Then: rank features by importance and interrogate the top one — ask out loud when its value is written, relative to when you need the prediction. Split by time whenever the data has time in it, and by entity whenever rows repeat per entity. Fit every transformation (scaler, imputer, encoder) inside the training fold only — and note that target encoding needs one level deeper than that, computed out-of-fold within the training data, or each row silently encodes its own label. Finally, write down for each feature the moment it becomes known; that document catches more leakage than any test.

Failure mode

Discovering it after launch. Leakage does not degrade gracefully — a model built on a leaked feature is not slightly worse in production, it is worthless there, because the column either arrives empty or arrives with a different meaning. The gap between validation and reality is the whole error and it appears on day one, which is one thing shadow mode is for: run the model on live inputs and compare its predictions to the offline ones. Note what that catches and what it does not. A feature that arrives empty or differently shaped at serve time shows up immediately; a bad split — temporal, group, duplicate — reproduces perfectly online, because the model is computing the same function on the same inputs. Those only surface when the outcomes land.