Large Language Models · entry 02/06

Tokens & tokenization

Models don't read letters or words — they read tokens, and half the puzzling behaviors and all of the pricing make sense once you do too.

What a token is

A token is a chunk of text — commonly a word, a word piece, or punctuation — drawn from a fixed vocabulary of tens or hundreds of thousands, learned by repeatedly merging the most frequent character pairs in the training corpus (byte-pair encoding). "the" is one token; "tokenization" might be token|ization; a rare surname might shatter into five fragments. English averages roughly four characters per token.

Why models seem weirdly bad at letters

Ask a model to count the r's in "strawberry" and it can stumble — not because it's stupid, but because it may never see letters: straw|berry arrives as two opaque ids. Spelling, rhyming, character arithmetic and exact string lengths are all tasks performed through a blindfold woven at tokenization time. The same applies to digits: numbers split into inconsistent chunks, which is part of why arithmetic wobbles.

Why you're billed in tokens

Every token in and out costs compute, so APIs price in tokens and context windows are measured in them. Practical consequences: verbose prompts are literally expensive; JSON with long keys costs more than terse formats; non-English text often tokenizes worse (more tokens per sentence — the corpus was English-heavy), so the same request costs more and fits less. Code fares well: popular languages tokenize densely.

One vocabulary, forever

The tokenizer is frozen before training and welded to the model — weights are meanings of those ids. It's also a quiet attack surface and bug source: adversarial strings that tokenize strangely, invisible Unicode that splits tokens, "glitch tokens" from corpus artifacts that send models sideways.

Failure mode

Prompt budgeting by eyeball. "About a page" is 300 tokens or 800 depending on language and formatting; truncation bugs and cost overruns hide in that gap. Count with the model's actual tokenizer — every provider ships one — and design formats (short keys, compact delimiters) with the meter running in mind.