Machine Learning Foundations · entry 09/10 · 3 min read

Calibration & uncertainty

A model that says 70% should be right 70% of the time — most aren't, the fix is cheap, and knowing what a model doesn't know is worth more than a point of accuracy.

Confidence is a claim

Every classifier emits a score, and everyone reads it as a probability. Calibration is whether that reading is earned: take every case scored 0.7 and check how many were actually positive. If it's 70%, the model is calibrated; if it's 45%, the number is decoration. This matters wherever the score feeds a decision rather than a ranking — a threshold, a triage queue, an expected-value calculation — because all of those do arithmetic on the number as if it meant something. Measure it with a reliability diagram (predicted vs. observed frequency, bucketed) and summarize it as expected calibration error: the population-weighted average gap between the two.

Why models drift out of calibration

Networks are commonly miscalibrated, usually toward overconfidence: trained to minimize loss on data they eventually fit well, they push probabilities toward the extremes. The direction is not a law, though — it depends on architecture and recipe, label smoothing and mixup overshoot into underconfidence, and several newer model families are well calibrated in-distribution without help. The classic fix is embarrassingly cheap — temperature scaling, a single parameter fitted on held-out data that softens or sharpens every output. It preserves the argmax and therefore top-1 accuracy, but it moves every fixed probability cutoff, so re-derive your thresholds after calibrating or you have shipped a different system. And it is an in-distribution fix: calibration fitted on a clean validation split decays under shift, so fit it on data that looks like deployment and re-check it there.

Two kinds of not-knowing

Uncertainty comes in two flavors worth separating. Aleatoric is noise in the world — two identical applications with different outcomes — and more data will not remove it. Epistemic is the model's own ignorance, high where training data was thin, and more data does fix it. The distinction is operational: epistemic uncertainty is what should trigger abstention and human review, because it flags the cases the model has no business deciding. Ensembles and their cheap approximations approach it through the disagreement between several plausible models — useful, and imperfect: members also disagree on genuinely noisy cases, and the converse does not hold. Models that share an architecture and a training set agree confidently on inputs none of them has any business judging, so agreement is not evidence that you are in distribution.

Language models make this harder

A generated "I'm about 90% sure" skews overconfident, for a familiar reason — preference training rewarded sounding certain — but it is not noise, and on preference-tuned models it is often better calibrated than the model's own token probabilities, which score how likely the wording was rather than whether the claim is true. Either way, the practical answer is behavioral: sample several times and measure agreement, check claims against retrieved sources, and make "not in the documents" a first-class allowed answer.

Failure mode

Treating an uncalibrated score as a probability in a business rule. "Auto-approve above 0.9" sounds prudent and is meaningless if 0.9 corresponds to 60% correct — the threshold was chosen against a number the model never promised. Before any score becomes a cutoff, plot it against observed outcomes; the shape of that curve decides where the line goes, and it is the cheapest chart in machine learning to produce.