AI Safety & Alignment · entry 04/05

Evaluating safety

Safety claims rest on evaluations — refusal tests, capability probes, red teams, and the frontier frameworks that tie safeguards to measured thresholds. Know what each can and cannot certify.

Two different questions

Safety evaluation splits into propensity — what does the model do unprompted or under ordinary pressure? — and capability — what can it do when someone tries hard? A model can be perfectly polite (low propensity) while capable of real harm if jailbroken, or incapable regardless of intent. The distinction decides the safeguard: propensity problems yield to training; capability concerns demand controls outside the model — access restrictions, monitoring, staged deployment — because a jailbreak can reach whatever exists underneath the manners.

The bench and the red team

Benchmarks scale: batteries of harmful requests scored for refusal, paired with benign look-alikes scored for over-refusal — both numbers matter, since a model that refuses medical questions "to be safe" has its own casualty count. But benchmarks age into targets, and static tests miss what determined humans find, so serious evaluation adds red teams: experts paid to break the model, increasingly assisted by automated attackers that mutate and search at machine speed. The asymmetry is permanent — evals prove presence of failure, never absence — which makes every safety number a lower bound on what's findable.

Frontier frameworks

For large-scale risks — weapons uplift, major cyber capability, autonomous replication — frontier labs publish responsible-scaling frameworks: defined capability thresholds, tests run during training and before deployment, and pre-committed safeguards (up to not deploying) when a threshold trips. The strength of this design is that it converts a vague promise into auditable if-then rules, written before the commercial pressure arrives. The honest caveats: thresholds are judgment calls, evaluation of a capability can lag its emergence, and the voluntary schemes bind only their adopters — though regulation has begun making parts of the pattern mandatory for the largest models. It is the best current answer, not a solved problem.

Failure mode

Treating the lab's evals as your evals. The model card certifies general behavior under the vendor's conditions; it says nothing about your system prompt, your tools, your retrieved documents, your users' incentives — the layer where most deployed risk actually lives. Teams inherit a safety report and skip the part where they test the product they built. Run your own harm cases through your actual stack, wired to your actual tools, and re-run them on every model swap: behavior differences between versions are large enough to flip your results.