The Lab · instrument 12/16 · interactive

Leak a model

A real logistic regression on a thousand loan applications, scored twice — once the way your validation set scores it, once against a future it has never seen.

What's real here

The model is a genuine logistic regression fitted by gradient descent, scored by rank AUC, retrained in your browser every time you touch a toggle. The split is by customer, so no applicant appears on both sides of it. Only the world is synthetic — and it is synthetic honestly: the collections column is generated from the outcome, because collections calls only happen after somebody defaults, and the neighborhood default rate is computed across the whole history, because that is how a careless team computes it. Nothing is faked to make a point. The leaks leak for the reasons real leaks leak.

Where to push

Start from the four application-time columns: the two numbers sit close together, both unremarkable. Now turn on prior defaults. Both rise, together. That is what a genuinely good feature looks like, and it is worth seeing first, because the reflex this page could otherwise teach — big jump means leakage — is wrong.

Then turn on collections calls. Validation leaps into the nineties. Production does not move. You have built a model that predicts default from evidence that only exists once default has already happened, and the validation set cheerfully agrees with you because it was drawn from the same finished history.

Now the subtler one. Turn collections off and zip default rate on. Validation climbs respectably — and production falls below where it started. This is target encoding computed over the whole dataset: with a handful of applications per postcode, each row's feature value is substantially its own label handed back to it. It does not merely fail to help; it displaces real signal with a number that will not exist at prediction time. Computed out-of-fold, so that no row can see its own label, the same technique is perfectly legitimate.

The picture that names the rule

Every column set you try is plotted on the right, validation against production. Honest configurations land on the diagonal; the ones falling below it lied to you, and how far below is how much. Nothing about the size of the validation score tells you which you have — only the distance from that line does, and you cannot measure that without something standing in for the future.

One caveat, and it matters: a gap is a symptom, not a diagnosis. If the world itself moves — a recession reprices risk, a policy change alters who applies — an entirely honest model falls below the line looking exactly like a leak. This page cannot show you that, because its only drift is a rising base rate, which rank-based scores ignore. The gap tells you something is wrong between the number you measured and the world you deployed into. Which of the two, you have to go and find out.

Failure mode

Having no production axis at all. Almost every team has the left half of this page and none of the right, which means a leak is invisible until it ships and the honest read is that the vertical position of your point is simply unknown. The substitutes are cheap and specific: split by time as well as by entity, hold out a period nobody touches, interrogate your top feature by asking when its value is written, and run in shadow mode before anyone depends on it.