AI Agents · entry 05/08 · 3 min read

Agent memory

Agents forget completely between sessions — persistent memory is an engineering system you build: what to store, when to retrieve it, and what to do when it turns out to be wrong.

There is no memory; there is a file

A model's state is its weights and its context window, and the window ends with the request. Every product that "remembers you" is doing the same three things: deciding what from this session is worth keeping, writing it somewhere, and selecting a slice of it back into a future window. That is a retrieval system with an editorial policy attached — the hard parts are the policy, not the storage.

Three kinds worth separating

Borrowing the cognitive vocabulary earns its keep here because the three behave differently. Episodic: what happened — past conversations, actions taken, outcomes. High volume, mostly stale, best stored raw and retrieved by search when a specific past event matters. Semantic: durable facts — the user prefers metric units, the production database is Postgres, this customer is on the enterprise plan. Low volume, high value, worth injecting into every window while that set stays small — past that, retrieve it just in time like anything else. Procedural: how to do things here — the runbook this agent learned the hard way. The failure of most memory features is treating all three as one pile of text and retrieving from it by similarity.

Writing is the hard half

Retrieval gets the attention; the write path decides whether the system is useful or radioactive. Extracting durable facts from a transcript means judging what generalizes — "prefers dark mode" does, "is annoyed today" does not — and doing it without a model enthusiastically recording every passing remark. Memories need provenance and a timestamp, because the fact that mattered in March is wrong in September, and they need an update path: a good memory system revises rather than accumulating contradictions. Give the user a way to see and delete what's stored, both because regulation increasingly requires it and because it is the only honest way to fix a wrong memory.

Failure mode

Poisoned memory, which is the compounding version of prompt injection. Text the agent read once — a webpage, a support ticket, an email — gets distilled into a "fact" and written down; from then on it is injected into every future window as trusted context, no longer traceable to the untrusted source it came from. One bad write outlives the session that produced it. Labels and provenance help and are not sufficient — the layer with teeth is architectural, exactly as it is for injection generally: make promotion into always-injected memory a privileged write, gated behind user confirmation, and never let a fact the agent inferred from content it fetched reach that tier on its own. Keep the store auditable, because an agent that remembers wrong is worse than one that remembers nothing.