AI Agents · entry 05/05

Evaluating & containing agents

Evaluating an agent means checking what changed in the world, auditing how it got there, and fencing what it can touch.

Grade the job, not the sentence

Response-quality metrics — helpfulness, tone, fluency — measure the wrong object once a model acts. The unit of evaluation is the task: given this starting state and this goal, did the world end up in the right condition? Build a suite of tasks with programmatically checkable end states — ticket closed with the correct resolution, bug actually fixed, refund issued exactly once — and measure end-to-end success. Everything in evaluating models still applies — held-out cases, honest baselines — plus a twist: the same task can succeed via a clean path or a lucky, ugly one, and only one of those generalizes.

Read the trajectory

The pass rate says whether; the trajectory says why. Log every tool call with its arguments and result, then audit: wrong tool chosen, malformed arguments, an error ignored, the same failing call retried six times, success declared without checking. Roll the audits into metrics worth tracking — steps per completed task, tokens per task, loop rate, recovery rate after a failed call. And keep rotating fresh tasks in: agents iterated against a fixed suite overfit it as surely as models overfit training data.

for task in suite:
    env = sandbox(task.fixture)        # fresh state, fake credentials
    log = run(agent, task.goal, env, max_steps=40)
    record(task.id,
           passed=env.state_matches(task.expected),
           steps=len(log.calls), errors=log.tool_errors)

Containment: sandbox, scope, gate

Because the model only touches the world through the runtime, the runtime is where the limits go. Sandboxing: development and evaluation run against fixtures and staging copies, never production. Least privilege: per-session, short-lived credentials scoped to the task — the research agent gets read-only keys, and no agent gets the org-wide admin token. Approval gates: actions that are irreversible or externally visible — payments, deletion, anything that emails a human — pause for sign-off, ideally with a dry-run diff of what is about to happen.

A junior operator, not an oracle

The working posture: a smart, tireless junior colleague in their first month. Real work, reviewed. Authority expanded per task type as the track record accumulates, not all at once. Logs sufficient to reconstruct any incident afterward. Review rates can fall as evidence comes in — but by measured risk tier, not by vibes after one good week.

Failure mode

Grading the story instead of the state. Agents narrate confidently, and the closing summary — "Done! All tests pass." — is itself model output, subject to hallucination like everything else. Suites that score the transcript reward agents that learned to sound finished. Verify the world instead: query the database, run the tests yourself, check that the file exists. An eval that believes the agent's self-report is measuring persuasion, not performance.