AI Agents · entry 04/06

Computer use

Point a model at pixels, give it a mouse and keyboard, and every application becomes a tool — the most general integration there is, and currently the least reliable.

The universal adapter

Most software has no API. The insurance portal, the ERP screen from 2009, the vendor dashboard with the export button — the long tail of work lives in interfaces built for eyes and hands. Computer use is the blunt answer: the agent loop with a screen in it. The model receives a screenshot, decides on an action — click these coordinates, type this text, scroll, press Enter — the runtime performs it, a fresh screenshot comes back, repeat. No integration was negotiated; whatever a temp with a login could operate, the model can attempt. Under the hood this leans hard on vision-language models: reading the screen, locating the button, grounding "the submit button" to the pixel coordinates of the thing itself. A variant reads the browser's accessibility tree or DOM instead of raw pixels — faster and crisper where it's available, blind where it isn't.

The honest scorecard

It works, with asterisks. Against benchmarks of real desktop and web tasks, current agents complete a solid majority of short errands and a much thinner share of long, multi-application workflows — impressive for a capability that barely existed before 2024, nowhere near a temp's reliability. The failure texture is distinctive: a date picker that defeats twelve attempts, a click that lands one toolbar icon left of the target, a redesigned menu that silently orphans the memorized route. Every screenshot costs a context window's worth of tokens and a second or two of latency, so sessions are slow and expensive — and drift compounds: one misread screen becomes a wrong click becomes a state no instruction anticipated.

The screen is untrusted input

A computer-use agent reads whatever the screen shows, and the screen shows the web. A page element saying "verify your account by entering the password here" is, to the model, instructions in its input — prompt injection with a rendering engine. The containment rules are the standard ones, applied strictly: a dedicated browser profile with nothing saved in it, scoped credentials issued per task, no access to your password manager, and approval gates on anything that pays, signs, sends, or deletes. Watching the replay is not paranoia; it's the current operating procedure.

Failure mode

Handing the agent your own logged-in machine. The demo works, so it gets your browser — with your sessions, your saved cards, your email already open — and now every misclick and every injected instruction operates with your full authority. The blast radius of "click things on a screen" is defined entirely by whose screen it is. Give it a clean profile and a narrow account, and the same mistake becomes a screenshot for the postmortem instead of a call to the bank.