AI Safety & Alignment · entry 07/09 · 3 min read

Interpretability

Behavioral testing tells you what a model did; interpretability tries to say why — features, circuits, and steering, plus an honest account of how far it currently reaches.

Why open the box at all

Everything else in this section judges a model by its outputs, and outputs have a ceiling as evidence: a system that behaves well under observation may behave differently elsewhere, and no eval proves absence. Interpretability aims at a different channel — reading the computation itself, so that "is this model doing what we think" becomes a question about mechanism rather than a question about test coverage. It is also plain engineering: knowing why a model fails is the shortest path to fixing it.

Features and the superposition problem

The natural unit to look for is a feature — an internal direction that means something, like "this text is legal boilerplate" or "the Golden Gate Bridge." The obstacle is that individual neurons are stubbornly polysemantic: one neuron fires for several unrelated things, because models pack far more features than they have dimensions (superposition). The tool that broke this open is the sparse autoencoder: train a wide, sparsely-activating layer to reconstruct a model's internal state, and its units line up with human-legible features far better than raw neurons do — though SAE features have since underperformed simple baselines on several downstream tasks, and the newest circuit work has moved on to transcoders. Millions have been extracted from frontier models; a much smaller set has been confidently named and — the part that makes it science rather than storytelling — causally tested: clamp one of those high and the model's behavior changes in the way the label predicts.

Circuits and the honest state of it

Above features sit circuits: small subnetworks that compute something specific, traced by patching activations between a clean and a corrupted run and watching what changes. Real ones have been mapped — induction heads that find an earlier occurrence of the current token and copy whatever followed it, pathways for simple factual lookup, the mechanics behind some multi-step reasoning. What is not true is that anyone can read a frontier model end to end. Coverage is partial, features found depend on the autoencoder you trained, and complete coverage by current dictionary-learning methods is estimated to cost far more compute than training the model in the first place. The honest summary: real, causal, accelerating, and nowhere near a complete account of a large model — which is why it complements behavioral alignment rather than replacing it.

Failure mode

Believing a satisfying story. A feature that lights up on your examples, a heatmap that matches intuition, a circuit diagram that explains yesterday's bug — all are hypotheses, and the field's recurring lesson is that plausible interpretations survive until someone runs the intervention. The discipline is causal: if the story is right, clamping or patching should change behavior in a predicted direction, on inputs you did not use to build the story — and note that plain ablation is the weakest version of that test, since backup components routinely compensate and hide a part's real importance. Without that test, interpretability output is an attention map with better branding.