Large Language Models · entry 04/06

Context windows & memory

A model knows its weights and its context window — nothing else. Everything called 'memory' is engineering that decides what to put in the window.

◈ generating · llms-context-windows-and-memory

The window is the world

At inference, a model sees exactly the tokens in its context — system prompt, conversation, retrieved documents, tool results — and nothing else. No recollection of yesterday's chat, no awareness of the next one. Within the window, attention connects everything to everything; one token past the edge might as well not exist.

What long context costs

Attention compares all pairs, so cost grows steeply with length, and serving keeps a KV cache — stored keys and values for every token — whose memory footprint is why long-context serving is expensive. Modern models advertise hundreds of thousands to millions of tokens, with two honest caveats: you pay for every token every call, and recall inside huge contexts isn't uniform — models can skim the middle ("lost in the middle"), so placement of critical facts still matters.

"Memory" is context management

Every persistent-memory feature is some flavor of: store information outside the model, select a relevant slice, write it into the next window. Chat history? Replayed into context. Long-running assistant memory? Notes written to storage and retrieved later — retrieval wearing a friendlier name. Agents that "remember" projects keep files and re-read them. The craft is curation: what to persist, what to summarize, what to drop.

Practical hygiene

Put stable instructions early (they also cache well — providers discount repeated prefixes); summarize stale conversation instead of replaying it; retrieve five relevant paragraphs rather than pasting the manual. A context assembled deliberately beats a bigger window used as a junk drawer — on cost, latency, and accuracy at once.

Failure mode

Silent truncation. Conversations grow until something — the client, the server, the framework — trims from the top, and the system prompt or the user's original goal quietly falls off the edge. The model doesn't error; it just drifts. If behavior degrades late in long sessions, audit what was actually in the window before blaming the model.