AI Infrastructure & MLOps · entry 05/05

LLMOps

Building on language models added a new operational layer: versioned prompts, eval harnesses as the test suite, token observability, and routing across models that change under you.

The configuration surface grew

A classic model deployment ships weights; an LLM product ships weights plus a system prompt, tool definitions, retrieval configuration, sampling parameters, and guardrail settings — each of which changes behavior as much as a retrain would. Treat them as production configuration: versioned, reviewed, deployed through the same pipeline as code, rollback-able in minutes. The team that hot-edits a prompt in a dashboard on Friday learns why this section exists on Saturday.

Evals are the test suite

There is no compiling an LLM product; there is only evidence. The working pattern is eval-as-CI: a graded suite of real cases — including every incident that ever embarrassed you — run automatically on any change to prompt, model, retrieval, or tools, with promotion gates on the scores. LLM-as-judge makes this affordable; periodic human calibration keeps the judge honest. The discipline matters double at model-swap time: providers deprecate and upgrade on their schedule, not yours — and the agent version of this is stricter still. Behavior differences between model versions are large enough to flip product decisions.

Observe the tokens, own the costs

LLM observability traces each request through its whole tree — retrievals, tool calls, retries, sub-agents — with token counts and cost attached, because latency and spend hide in the tree, not the final call. Cost control is then routing: cache aggressively (prefix caching for shared prompts), send easy traffic to small models and hard traffic to frontier ones, cap runaway loops, and set budgets that alert before finance does. Privacy sits in the same layer: what leaves your boundary, to which vendor, retained how long — an architecture question answered before the first integration, or painfully after.

Failure mode

Playground-to-production teleportation. It worked in the console, so it ships: unversioned prompt, temperature nobody recorded, no eval baseline, no cost projection. Three weeks later the provider updates the model, output format shifts, a parser breaks, and the team can't say what changed because nothing was pinned. The whole discipline of this article is the antidote to one habit: treating a probabilistic dependency like a stable library. It's a dependency that drifts — operate it like one.