AI Safety & Alignment · entry 02/09 · 3 min read

How models are aligned

RLHF, constitutional AI, and system-level policy — the current toolkit that turns a raw text predictor into something you can hand to the public, and what each layer can't do.

Layer one: preference training

The workhorse is the third stage of the training pipeline: generate candidate answers, have raters pick the better one, train a reward model on those picks, optimize the model against it. Helpfulness, tone, refusal style, admitting uncertainty — this is where they're shaped. It inherits its raters: their guidelines, their attention spans, their blind spots, and their preference for confident, agreeable-sounding answers, which is how sycophancy gets trained in while everyone is trying to train it out.

Layer two: AI feedback and constitutions

Human rating doesn't scale to millions of judgments, so labs point models at the problem: the model critiques and revises its own outputs against a written constitution — an explicit list of principles — and candidate answers are ranked the same way, with training on the result (constitutional AI; RLAIF, more broadly). The gains are scale, consistency, and inspectability: a constitution is an artifact you can read, argue with, and version, unlike ten thousand raters' moods. The cost is recursion: the judge is itself a model with exploitable quirks, and principles conflict in ways a document doesn't resolve — "be helpful" and "be harmless" collide daily, and the resolution is learned, not specified.

Layer three: deployment policy

Above the weights sits everything operators add: system prompts declaring role and rules, input and output filters, tool permissions, rate limits, human approval gates for consequential actions. This layer is the cheapest to change — policy updates ship in hours, retraining ships in months — which makes it the right home for anything specific to your product. It's also the layer adversaries attack first, because text instructions are suggestions to a text predictor, not constraints on one.

What the stack achieves — and doesn't

Together these layers move behavior enormously: compare any raw base model to its assistant version and the difference is night and day. But each layer biases a distribution rather than deleting capability; the base model's knowledge and habits remain underneath, reachable by unusual paths. Alignment as practiced is defense in depth over a statistical object — genuinely effective, never absolute — which is why measuring it is its own discipline, and why claims of "cannot" deserve translation to "did not, under the conditions we tested."

Failure mode

Confusing the system prompt for the safety system. Teams write "do not reveal the following" above a secret and consider the matter closed; the first curious user extracts it by asking politely in Portuguese. The prompt layer sets defaults for cooperative inputs; adversarial robustness has to come from the layers with actual enforcement power — what the model can access, what actions require approval, what the surrounding code refuses to execute. Put policy in prompts; put limits in architecture.