AI Safety & Alignment · entry 03/09 · 3 min read

Bias & fairness

Models inherit the world's patterns and a pipeline's choices, then apply them at scale — the mechanisms, the metrics that provably cannot all hold at once, and what actually reduces harm.

Where it comes from

Bias in a model is rarely one villain; it accumulates. The training data records a world with historical patterns in it, and a model trained to predict that data predicts those patterns faithfully. Sampling adds its own skew — who is represented, in what proportion, photographed by which cameras, writing in which languages. Labels carry the guidelines and assumptions of whoever wrote them. And the target variable is often a proxy: predicting "who gets arrested" is not predicting "who commits crimes," and a model optimizing the proxy inherits every distortion between the two. That last step is the one teams skip, and it is usually the largest.

The impossibility that ends most arguments

"Make it fair" is under-specified because fairness has several formal definitions that are mutually incompatible. Demographic parity wants equal selection rates across groups. Equalized odds wants equal error rates — true-positive and false-positive alike (relax it to the true-positive half and you have equal opportunity). Calibration, or its cousin predictive parity, wants a predicted 70% to mean 70% within each group. When base rates genuinely differ between groups, it is provable that you cannot satisfy all three at once — this is not an engineering gap but a mathematical result. The COMPAS recidivism dispute is the canonical demonstration: ProPublica showed unequal false-positive rates, Northpointe showed equal precision per score band, and both were right about the metric they cited. So the work is not to find the fair metric; it is to decide, explicitly and with the people who bear the consequences, which error the system is allowed to make more often — and to write that decision down where an auditor can read it.

What actually helps

Measurement first: performance disaggregated by group, on data that represents deployment, because a strong average routinely hides a weak slice — this is the same discipline as any other evaluation, applied where it is least comfortable. Then the levers, roughly in order of leverage: fix representation and labels upstream; reconsider the proxy; adjust the operating point with the tradeoff stated — where law permits, since differentiated cutoffs by protected group are prohibited outright in US employment testing; add abstention so uncertain cases route to a human. For generative systems the equivalents are audit sets probing stereotype and refusal asymmetries, plus preference training against them. None of it is one-time work; drift moves the numbers.

Failure mode

Removing the protected attribute and declaring the problem solved. The variable disappears; its correlates — postcode, school, name, purchase history, phrasing — remain, and a capable model reconstructs the signal without ever being told it. Blindness is not fairness; it is fairness you can no longer measure, which is worse, because the disparity still happens and now nothing in your pipeline can see it. Keep the attribute for measurement, guard it as sensitive data, and audit outcomes rather than inputs.