Large Language Models · entry 06/06

Scaling laws & small models

Loss falls on a smooth curve as models grow — abilities arrive in jumps anyway, and the same curves now tell you when a small model is the right answer.

The smooth part

Across many orders of magnitude, next-token loss falls as a clean power law in parameters, data and compute — smooth enough that labs fit the curve on small runs and budget nine-figure training runs against the extrapolation. A second result reset the field's habits: for a fixed compute budget, most early models were too big and undertrained; compute-optimal training wants parameters and data grown together, which is why "smaller model, far more tokens" became the recipe.

The jumpy part

Loss improves smoothly; capabilities often don't. Multi-step arithmetic, translation, tool use — each looked absent, then present, across a scale threshold. Some of that jumpiness is measurement artifact (all-or-nothing tests make gradual progress look sudden), but the practical lesson survived: you cannot fully list what the next scale-up will be able to do, which is precisely why capability evaluation became its own discipline.

Scale stopped being the only axis

Two other curves now carry weight. Data quality: curated and synthetic data lets small models punch far above their weight class. Inference-time compute: letting a model reason longer — extended chains of thought, multiple samples, search — buys accuracy with serving compute instead of training compute, and for hard problems buys more than the next size class does. "Bigger" is now one lever on a panel.

When small is right

Frontier models cost frontier prices per token. A distilled or small open model wins when the task is narrow (classification, extraction, routing, one domain's paperwork), latency or privacy demands local serving, or volume makes unit cost the whole game. The honest method: build the eval first, then walk down the size ladder until quality breaks, and serve one rung above that.

Failure mode

Extrapolating ability from loss. The curve promises the model gets better at predicting text; it does not say your task's threshold is near, or that ten times the spend buys ten times the usefulness. Benchmarks saturate, tasks bottom out, and the marginal token of scale is only worth what your eval says it is.