The Lab · instrument 10/16 · interactive

Route to the experts

A real routing loop: tokens pick their top-k experts, winning makes an expert likelier to win again, and without a thumb on the scale a few experts take everything.

What's real here

Each expert holds an affinity vector, each token is scored against all of them, and the top-k win — the routing step from beyond the dense transformer. Winning then nudges that expert toward the kind of token it just served, which is the whole mechanism: routing is learned, so early advantages compound. The balancing slider implements the scheme real trainers use — a per-expert bias pushed up when an expert is under-used and down when it is hogging traffic.

Where to push

Leave balancing off and wait. The bars diverge, the dashed even-split line stops meaning anything, and several experts go starved — their parameters still sitting in memory, still paid for, doing nothing. That is router collapse, and it is not a bug in this page: it is what an unconstrained learned router does, which is why every production mixture-of-experts applies balancing pressure of some kind. Now drag the slider up. The bias term pushes back on whoever is winning, the bars converge on the even line, and the starved experts come back. Then drop top-k to 1 and watch collapse arrive faster — fewer winners per token means a sharper rich-get-richer race.

Failure mode

Reading "total parameters" as capability. The panel shows what the model actually spends per token — with 32 experts and top-2 routing, about 6% of the parameters do the work while 100% of them occupy memory. That is the real trade: capacity bought with memory, not with compute. And when the router collapses, you are paying memory for experts that never run, which is the most expensive way possible to own a parameter.