The Lab · instrument 13/16 · interactive

Break an agent

An agent loop with three architectural legs and two defenses — the lethal trifecta as something you can turn off, next to the arithmetic that decides whether the task finishes at all.

What's real here

No model runs on this page. This is an architecture simulator, and it is worth being precise about which parts are which. The taint rule is real and standard: anything derived from untrusted text is itself untrusted, and no amount of downstream processing launders it. The arithmetic is plain multiplication — a step that works 95% of the time, done twenty times, finishes 35.8% of the time — though it assumes steps fail independently and equally often, which real loops do not: a bad plan poisons everything after it, and frameworks retry. What is assumed is that the model obeys the planted instruction, which models demonstrably do often enough to matter.

Where to push

Start with the defaults: a support-triage agent that reads a ticket, opens a page linked from it, looks up the account, and emails a reply. Both the ticket and the page are written by someone outside your company, which is why both count as untrusted. Watch the taint spread forward from the step that read the payload and arrive at the send. Then switch the planted instruction to blunt and notice the filter catch it — and notice that the polite version, which no filter is confident about, is the one that works. Filtering raises cost. It does not close the surface.

The two defenses do different jobs

Scoped credentials do not stop the leak; they shrink it. The lookup returns one customer's record instead of the table, the injection still steers the send, and one record still leaves. That is a real and worthwhile reduction in blast radius, and it is not a fix. Separate read from act is the one that closes the path: the component holding the credential never reads the page, so private data never joins the tainted context, and the email goes out with parameters the attacker could not touch. Now turn off the send tool alone and read the verdict carefully — an agent that can fetch a URL already has an outbound channel, because the data can ride out in the query string. Removing the send tool removes a tool, not a leg.

Then drag the step count

The lower panels are the other failure, and it needs no attacker. At seven steps and 95% the picture looks survivable; drag to twenty-five and under a third of runs finish. The curve is exponential and intuition is linear, which is why per-step accuracy is a misleading number to report. Push reliability to 99% and drag to forty: a five-times-better step still leaves only two runs in three completing. Long autonomy buys distance, not safety.

Failure mode

Turning one toggle off and calling it solved. The trifecta breaks only if a leg is genuinely absent, and legs come back quietly: read access added for a roadmap item, a webhook added for an integration, a credential widened because a ticket was blocked. Each is reasonable alone and the third one re-arms everything. The audit is of the combination, it has to re-run whenever the combination changes, and the only answers that survive contact are the ones that hold when the model does exactly what the attacker asked.