Machine Learning Foundations · entry 05/05

Evaluating models

Accuracy lies, benchmarks saturate, and the eval you didn't write is the failure you'll ship — measurement is where ML rigor lives or dies.

Accuracy lies first

A fraud model that flags nothing is 99.7% accurate when 0.3% of transactions are fraud. Class imbalance makes accuracy a compliment, not a measurement. Precision (of the flags, how many were right) and recall (of the frauds, how many were caught) pull in opposite directions, and which direction is expensive is a business fact, not an ML one: a missed cancer and a false alarm do not cost the same, so no single score can decide for you.

Benchmarks age like milk

Public benchmarks are useful the way standardized tests are: comparable, gameable, and eventually taught to. Once a benchmark becomes a target — or leaks into training data — climbing scores stop meaning climbing ability. Treat leaderboard deltas as marketing until reproduced on data the model provably hasn't seen, ideally yours.

Evaluating generative systems

There is no ground-truth answer sheet for "summarize this contract." The working toolkit: golden sets with rubrics, pairwise preference (is A better than B — humans agree with each other more on comparisons than on scores), and LLM-as-judge, which scales cheaply but inherits the judge's biases — verbosity preference, position bias, self-preference — so calibrate the judge against human labels before trusting it.

The eval is the spec

In practice the evaluation set does the job a specification used to: it defines what "working" means, catches regressions when you swap models or prompts, and encodes the failures you've promised not to repeat. Every production incident should end life as an eval case. Teams that treat evals as the product's test suite ship; teams that eyeball outputs ship surprises.

Failure mode

Goodhart's law, always: when a measure becomes a target, it ceases to be a good measure. The model optimizes the metric, the team optimizes the benchmark, the vendor optimizes the demo — three layers of proxy between you and "does it actually work." Keep one eval nobody optimizes against, and read real transcripts weekly; the number is a summary, not the thing.