The Lab · instrument 03/09 · interactive

Train a tokenizer

Byte-pair encoding, the algorithm behind every frontier model's vocabulary, running on whatever text you type — merge by merge.

What's real here

This is the actual byte-pair-encoding loop from tokens & tokenization: count every adjacent pair of symbols in the corpus, weld the most frequent pair into a new symbol, repeat. Production tokenizers run the same loop over terabytes with a few tens of thousands of merges; this one runs it over your paragraph with a slider. The merge log is the vocabulary's autobiography, in order.

Where to push

Pull the merges slider to zero: everything is characters, and your sentence costs a fortune in tokens. Slide up and watch the economy improve — t + h, then th + e, and suddenly the most common word in English is one atom. Now edit the corpus. Delete every "the" and watch the vocabulary reorganize around different frequencies. Paste in code, or German, or your own writing, and see which words earn single tokens — frequency is the only merit that counts here, which is why a tokenizer is a census of its training data.

The part that bites

Tokenize a word the corpus never saw — a name, a typo, a technical term. It shatters into fragments, each one a token you pay for and the model must reassemble meaning from. This is why rare names get mangled, why non-English text often costs more per sentence, and why context budgets vanish faster than a character count suggests. The model doesn't see your text; it sees what the tokenizer did to it.

Failure mode

Forgetting the weld is permanent. A production tokenizer is frozen before training and fused to the weights forever — every quirk you just produced with the slider, some real model is living with at scale. When a model stumbles on spelling, arithmetic, or an unusual string, check what the tokenizer handed it before blaming the intelligence: half the mystery usually dissolves right there.