section 10 · status: live · 9 entries · updated 2026-08-29
AI Safety & Alignment
Models do what the training signal rewarded, which is not always what anyone wanted. The gap has a name, an engineering practice, an adversary, harms that need no adversary at all, an evaluation discipline, and a genuinely open long game. Read top to bottom if you are new to it.
live · preference training reshaping an output distribution — the tail thins, never to zero
- What alignment means3 minthe gap between what you optimized and what you meant
- How models are aligned3 minpreferences in, constitution over, refusals shaped
- Bias & fairness3 minthe harm that arrives without an attacker
- Jailbreaks & prompt injection2 minthe model does what the text says — whoever wrote it
- Privacy & memorization2 minthe model remembers more than anyone intended
- Evaluating safety2 mina passed eval is a lower bound, not a verdict
- Interpretability3 minreading the weights instead of trusting the output
- When it goes wrong3 minthe incident response nobody drills until they need it
- The long game3 minoverseeing systems smarter than the overseers
check yourselfAnswer before you open
Trying to recall something teaches it better than re-reading does. Have a go, then open the answer.
A model refuses your harmful request. What have you learned about its capabilities?
Almost nothing. Refusal is trained propensity, not absence of capability — the base model's abilities remain underneath and unusual paths can reach them. Propensity yields to training; capability needs controls outside the model. Evaluating safety →
You removed the protected attribute from your model's inputs. Is it fair now?
No — its correlates remain, and a capable model reconstructs the signal without being told. You have removed your ability to measure the disparity, not the disparity. Keep the attribute for measurement and audit outcomes. Bias & fairness →
Your AI feature has been wrong for eleven days and nothing alerted. What should have caught it?
Distribution monitors on outputs rather than errors — refusal rate, answer length, citation rate, schema-validation failures — plus a held-out eval running continuously against production config, and a reporting path a human actually reads. When it goes wrong →
What does it mean to say a model is 'misaligned'?
Not that it is malicious. It means the behavior the training signal rewarded differs from what the designers meant — the objective was a proxy, and the model optimized the proxy faithfully. That gap is an engineering fact about specification, which is why it does not go away by making the model smarter. What alignment means →
Why can't a lab simply filter jailbreak prompts and be done?
Because the input space is unbounded and the attacker sees your filter and iterates while you do not see theirs. Filtering raises cost; it never closes the surface. The layers with teeth are architectural — least privilege, scoped credentials, and separating the code that reads untrusted content from the code that acts with secrets. Jailbreaks & prompt injection →
Your model was RLHF'd, red-teamed and passes its safety evals. What have you established?
A lower bound, and only on what you tested. Each alignment layer does something specific and none of them is a guarantee: preference training shapes a distribution and thins a tail without emptying it, and an evaluation reports the capability someone thought to look for. A passed eval is evidence of absence only in the places you looked. How models are aligned →