Large Language Models · entry 10/11 · 3 min read

Beyond the dense transformer

Mixture-of-experts broke the link between a model's size and its cost per token, and attention alternatives are chipping at the quadratic — the two architectural moves that actually stuck.

The dense assumption

In the architecture from transformers & attention, every parameter participates in every token: a 400-billion-parameter model does 400 billion parameters' worth of arithmetic per token, whether the token is a comma or a proof step. That coupling made "bigger" and "more expensive to run" the same decision. Both major architectural changes since are attempts to break a coupling like that one.

Mixture of experts

Replace each feed-forward block with many parallel "experts" and a small router that picks a handful of many per token — commonly eight of a few hundred, often alongside an always-on shared expert. Total parameters go up; active parameters per token stay small — a model can hold hundreds of billions of weights while doing the arithmetic of a much smaller one, which is why nearly every frontier model is now sparse. The costs are real and specific. All those weights still occupy memory, so MoE trades memory for compute rather than making models cheap outright. Training is fiddlier: routers collapse toward favourite experts unless load-balancing pressure is applied. And serving is a distributed-systems problem — experts are spread across devices, so every token becomes a small routing and communication decision — and a batch's tokens scatter across hundreds of experts, leaving each one a slice too thin to be anything but bandwidth-bound. That is why MoE models can show worse latency than their active-parameter count suggests.

Chipping at quadratic attention

The other cost is attention's all-pairs comparison, quadratic in sequence length. The successful responses have been engineering rather than replacement — memory-efficient exact attention (the FlashAttention lineage), which removed the quadratic memory without touching the math; plus grouped-query and sliding-window variants that shrink the KV cache serving actually pays for, the second of which does cut the quadratic itself. The more radical alternatives — state-space models like Mamba, and linear attention — hold a fixed-size state and run in linear time, with genuinely strong results and one persistent weakness: exact recall of arbitrary detail from long context, which attention gets by construction — and pays for with a KV cache that never stops growing. The practical outcome is hybrids, interleaving a few full-attention layers among many linear ones, keeping most of the speed and most of the recall.

Failure mode

Comparing models by parameter count across architectures. A sparse 400B model with 30B active is not comparable to a dense 400B model on capability, cost, or hardware requirements — and it is not comparable to a dense 30B either. Parameter counts stopped being a unit of measurement once the field went sparse; what a claim needs is active parameters, total parameters, and measured performance on your own eval. Anything else is comparing two different quantities that share a name.