section 13 · status: live · 9 entries · updated 2026-08-29
AI Infrastructure & MLOps
Models get the headlines; infrastructure decides what ships. Why GPUs won and memory rules, how one network trains on ten thousand chips, the per-token economics of serving, what it costs the grid, what changes when the model runs on your own hardware, and the operational discipline — classic MLOps, the new LLMOps layer, and the security seams between them.
live · continuous batching — requests join mid-flight, the GPU never waits
- The hardware layer3 minmatrix multiplies, memory walls, and very expensive plumbing
- Training at scale2 minone loss curve, ten thousand chips, months of not crashing
- Serving & inference2 minthe meter runs per token — engineer accordingly
- Local & edge inference2 minthe model on your own silicon, no meter running
- What it costs3 minunit economics before the invoice teaches them
- What it costs the grid3 minthe energy question, with the arithmetic actually done
- The ML lifecycle2 minthe model is 5% of the system; this is the other 95%
- Securing the stack3 minthe attack surface that isn't the model
- LLMOps2 minprompts are config, evals are CI, tokens are COGS
check yourselfAnswer before you open
Trying to recall something teaches it better than re-reading does. Have a go, then open the answer.
Why is generating a token slower than reading one?
Prefill processes the whole prompt in parallel and is compute-bound; decode emits one token at a time and must stream the model's weights from memory for each, making it bandwidth-bound. That asymmetry sets both latency numbers you care about. Serving & inference →
Your AI feature's costs came in five times the estimate. Name three likely causes.
One request meant several calls (retrieval, rerank, generate, validate, retry); conversation length grew cost faster than linearly because every turn re-reads the transcript; and the long tail of hard requests cost far more than the median you modeled. What it costs →
You validated a model, then shipped the 4-bit version. What did you skip?
Evaluating the artifact you actually deployed. Quantization degrades non-uniformly — math, long context and low-resource languages suffer most — and generic benchmarks under-sample exactly those slices. Local & edge inference →
Why is memory, rather than raw arithmetic, usually the binding constraint when serving a model?
Because generating each token requires reading the model's weights, and moving those bytes is slower than multiplying them. Modern accelerators can do far more arithmetic than their memory can feed, so most serving tricks — quantization, batching, KV caching — are memory tricks first and arithmetic second. The hardware layer →
A model that passed every eval starts producing worse answers three months after launch, with no deploy. What are the candidates?
The world moved and the data did — new products, new phrasing, new user population — while the model stayed fixed. Or an upstream dependency changed under you: an unpinned model version, a re-embedded index, a retrieval config edit. This is why the lifecycle ends in monitoring, and why every input to an answer needs a version. The ML lifecycle →
Someone quotes you the energy cost of one AI prompt. What do you ask?
Which model, on which hardware, on whose grid. Published per-prompt figures span a factor of a hundred, mostly because they are not measuring the same event, and the answer is marginal, regional and time-varying — grid carbon intensity alone spans well over an order of magnitude between regions. Note too that at scale inference outweighs training, so the training run is the smaller half of the story. What it costs the grid →