AI Infrastructure & MLOps · entry 03/05

Serving & inference

Inference is where AI meets its cost of goods: prefill and decode, the KV cache, batching tricks, and the latency arithmetic that quietly designs your product.

Two phases, two economics

Serving a language model splits into prefill — ingest the prompt, all tokens in parallel, compute-bound and fast per token — and decode — generate the reply one token at a time, each step streaming the model's billions of weights from memory again, bandwidth-bound and comparatively slow. The two phases explain the two latency numbers that matter: time-to-first-token (prefill, plus queue) and time-per-output-token (decode). Long prompts tax prefill; long answers tax decode; and the KV cache — stored attention state for every token in context — is the memory hog that limits how many conversations a GPU can hold at once.

The serving bag of tricks

Modern inference stacks stack optimizations. Continuous batching: requests join and leave the batch mid-flight, keeping the GPU full instead of waiting for the slowest reply to finish. Paged KV caches: treat attention memory like virtual memory, ending fragmentation. Quantization: 8- or 4-bit weights cut bandwidth per token — often with negligible quality loss, measured, not assumed. Speculative decoding: a small draft model proposes several tokens, the big model verifies them in one parallel pass — same output distribution, fewer big-model steps. Prefix caching: identical prompt prefixes (your system prompt, few-shot examples) computed once and reused, which is also why providers discount cached input tokens and why stable prompt prefixes are a cost feature, not just a style choice.

Cost shapes the product

Per-token economics leak into design everywhere: caps on output length, tool results trimmed to the fields that matter, stale turns dropped from context rather than replayed forever, small models routed the easy traffic with the frontier model reserved for the hard tail. Teams that treat inference cost as a design input from day one ship sustainable products; teams that discover it in the first invoice redesign in public.

Failure mode

Benchmarking latency at concurrency one. The demo feels instant; production adds queueing, batch scheduling, cache pressure, and the p99 request arrives behind someone's 90,000-token prompt. Serving performance is a distribution under load, not a number in a quiet terminal: load-test at realistic concurrency and prompt mix, track p95/p99 of both latency phases, and decide explicitly what degrades when traffic spikes — because something will, by design or by accident.