Applied & Classical ML · entry 05/06 · 3 min read

Imbalance & anomalies

Fraud, defects, intrusions and rare disease share a shape — almost no positive examples, an adversary or a drifting world, and an accuracy score that is actively misleading.

The shape of the problem

When positives are 0.1% of the data, accuracy is worthless — a model predicting "never" scores 99.9% — and so is most intuition built on balanced problems. Precision and recall are the working vocabulary, and the honest summary is the precision-recall curve rather than ROC, which looks reassuring on extreme imbalance because a huge negative class makes the false-positive rate small no matter what. One caveat comes with that: a PR curve moves when the base rate moves, so it is not comparable across datasets, or across quarters, without saying so.

Two different tasks that get confused

Supervised rare-event detection: you have labels, just very few. Here the answer is ordinary classification with care — class weights or focal loss, threshold tuning against real costs, and evaluation on the metric that matters. Unsupervised anomaly detection: you have no labels at all and must define "unusual" — isolation forests and density estimates, which tolerate a contaminated training set, or one-class methods and reconstruction-error autoencoders, which assume you can hand them a clean sample of normal and are better called semi-supervised. These answer different questions, and the second family has no notion of which anomalies you care about. A machine that is merely unusual is not the same as a machine that is failing.

Resampling, and its honest limits

Oversampling the minority, undersampling the majority, and synthetic methods like SMOTE all rebalance the training distribution, and the evidence for them is worse than their popularity suggests. Studies on real clinical and tabular tasks keep finding no improvement in ranking — the same AUC as leaving the data alone — while reliably pushing predicted probabilities toward the minority class, so any score that comes out needs recalibration before it becomes a threshold. The benefit that does exist is largely confined to weak learners. What people actually wanted from resampling is available directly and without the distortion: keep the imbalance, and move the operating point against the cost of each error type.

The adversary and the drift

Fraud is different from rare disease in one decisive way: someone is watching your model and adjusting. A deployed detector changes the attacks it will see next month, so performance decays for reasons no retraining schedule anticipates. The operational answers are unglamorous — monitor the score distribution rather than only the accuracy, retrain on recent data often, keep human review in the loop for the boundary cases, and accept that the label you get back is delayed by however long a chargeback takes.

Failure mode

Ignoring the cost of the false positives. Push recall up and you also freeze real customers' cards, flag healthy patients for biopsy, and alert an on-call engineer at three in the morning until they stop reading alerts. Every alert has a price paid by someone, and whether nine true positives per hundred alerts is excellent or unusable depends entirely on what a review costs and what a miss costs — which is the calculation, not a detail of it. Choose the threshold from the cost of both errors, out loud, including who bears each one.