AI Safety & Alignment · entry 08/09 · 3 min read

When it goes wrong

AI systems fail silently, at scale, and often without an error — detection, containment, disclosure and the postmortem, adapted for failures that produce no stack trace.

Why the usual playbook doesn't fit

Software incident response assumes a signal: an exception, a latency spike, a failed health check. The characteristic AI failure has none of those. Plenty of AI incidents are loud and ordinary — timeouts, rate limits, context overflows, malformed tool arguments — and those you already know how to handle. The ones that hurt are the quiet kind: the system returns 200, the output is fluent, the dashboards are green, and thousands of answers were subtly wrong — a retriever whose recall collapsed, a prompt edit that changed refusal behavior, a model version that shifted tone. Mean time to detection is the whole problem, and it routinely runs into weeks at organizations that never planned for it.

Detection: instrument the things that don't throw

What actually catches these: output monitors that watch distributions rather than errors — refusal rate, answer length, citation rate, tool-error rate, the fraction of responses failing schema validation — with alerts on change as well as absolute floors, since a 40% refusal rate is bad regardless of yesterday's. A held-out eval suite run continuously against production configuration, so a regression is caught by a robot at 3 a.m. rather than a customer in a week. And the cheapest signal of all: a working path for users to report a bad answer, read by a human weekly. More incidents than anyone likes to admit were visible in someone's inbox before they were visible in a metric.

Containment, then disclosure

Contain in the order the blast radius demands: revoke the agent's tools before you debug the agent; roll back the prompt or model version before you diagnose the change; move the autonomy dial down to suggest-only rather than switching the feature off, if that keeps users working. Because prompts, model versions and retrieval indexes are versioned configuration, rollback should take minutes through the same reviewed pipeline as any deploy — if reverting a prompt needs a source change and a full build, that is itself a finding. Then the uncomfortable part: notify the people whose decisions were affected. AI errors are quiet, so the affected party usually cannot tell they were harmed without being told. For high-risk systems, reporting serious incidents is separately becoming a regulatory duty — but that duty runs to authorities, not to the person harmed, so telling them remains your call.

Failure mode

The postmortem that ends at "the model hallucinated." That is a description, not a cause, and it produces the non-action item "add to the prompt: be accurate." Push to the system: what let a wrong answer reach a user unreviewed? Why did nothing detect it for eleven days? What was the actual trigger — a retrieval regression, an unpinned model upgrade, a context that silently truncated? Then close the loop the way the discipline demands: every incident becomes an eval case, permanently. A team whose eval suite is a museum of its own outages is a team that stops having the same outage — and the discipline generalizes, because the harder version of this problem is the same question asked about systems nobody can review directly.