AI Infrastructure & MLOps · entry 05/08 · 3 min read

What it costs

Per-token pricing, the cost drivers nobody models, and the build-versus-buy arithmetic — the spreadsheet that decides whether an AI feature survives contact with finance.

The unit is the request, not the token

Token prices are the input; the number that matters is cost per completed task, and the gap between them is where budgets die. One user request rarely means one call: a retrieval step, a rerank, the main generation, a validation pass, a retry when validation fails. An agent multiplies that by its loop count, and because every turn re-reads the accumulated transcript, cost grows super-linearly with conversation length rather than linearly. Model the whole path before quoting anyone a number, and measure it on real traffic, where the long tail of hard requests costs several times the median.

The levers, in order of leverage

Routing first: send easy traffic to small models and keep the frontier model for the hard tail — the single biggest reduction available, often without measurable quality loss. Then caching: stable prompt prefixes discount input steeply, and a semantic cache over repeated questions removes the call entirely — with a tight similarity threshold and a short TTL, since a loose one serves stale or other people's answers. Then output discipline: capping length and demanding terse structured output cuts the expensive decode phase directly. Then reasoning budgets, spent only where they pay. Batch anything asynchronous — providers discount it heavily. Fine-tuning appears here too: a tuned small model can beat a prompted large one on cost per task, once volume amortizes the training.

Build, buy, or host

API pricing is a rental with no capital cost, no ops burden, and access to new models as they ship — at the price of a deprecation calendar you don't control; it wins decisively until volume is large and stable. Self-hosting open weights inverts that — hardware or reserved capacity, an ops team, your own evaluation and safety layer — and starts winning when utilization is genuinely high, when data cannot leave your boundary, or when you need a model nobody will deprecate on you. The honest comparison includes engineer-hours and idle GPUs, and the honest schedule assumes per-token prices keep falling, which they have, persistently — though cost per completed task has not fallen as fast, because more of the answer is now bought with test-time compute. A plan whose viability depends on today's rates gets better with time; one that depends on today's capabilities is a bet.

Failure mode

Pricing the demo. A feature is estimated on the happy path — one call, short prompt, tidy answer — ships, and meets reality: users paste enormous documents, agents loop, retries fire on malformed output, and one enthusiast discovers the thing and runs it ten thousand times. Costs land three to ten times the estimate. Instrument tokens per request from day one, alert on the tail rather than the mean, and put hard caps in the code — because the fastest way to learn your unit economics is a bill nobody budgeted.