AI Infrastructure & MLOps · entry 04/08

Local & edge inference

Open weights, aggressive quantization, and NPU-equipped consumer hardware made serious models run on laptops and phones — the real tradeoffs, and where local genuinely wins.

Why this became possible

Three curves crossed. Open-weight models got good — the gap to frontier APIs narrowed from "different sport" to "one weight class" for many tasks. Quantization matured: weights compressed to 4 bits cut weight memory roughly fourfold — somewhat less in practice, and not at all for the KV cache — with losses that are small and, crucially, measurable. And consumer silicon grew tensor hardware — unified-memory laptops that fit mid-size models whole, NPUs in phones — while a vigorous open-source serving stack (the llama.cpp lineage and its GGUF format, plus vendor runtimes) made "download and run" a single command. A capable assistant on a machine you own, offline, is now an ordinary Tuesday.

The honest tradeoff table

Local buys four things: privacy for your inputs (prompts and documents never leave the device — the deployment half of the privacy problem; it does nothing about what the weights already memorized), offline operation, near-zero marginal cost per token, and immunity to vendor deprecation. It pays in three: capability ceiling (the biggest models still don't fit), speed (consumer memory bandwidth serves tokens at reading pace, not API pace), and operations — you are now the platform team, patching runtimes and validating quantizations. Batteries and thermals referee the phone version of the argument.

Where local wins on merit

Not everywhere, but somewhere specific: regulated or sensitive data that must not leave the building; high-volume narrow tasks (small-model territory) where API margins dwarf electricity; latency-critical loops with no round-trip budget; fieldwork without connectivity; and products whose pitch is precisely "your data stays yours." The mature pattern is hybrid: a local model for the frequent, private, easy 80%, escalating the hard tail to a frontier API — routing, again, deciding the economics.

Failure mode

Validating the full-precision model and shipping the quantized one. The 4-bit file that fits on the laptop is not the model the benchmark measured; most tasks survive the squeeze, but some — math-heavy, long-context, low-resource languages — degrade non-uniformly, and the damage hides in exactly the slices generic benchmarks under-sample. Evaluate the artifact you deploy, at the quantization you deploy, on your own eval set — the zero-cost token is only cheap if it's still right.