Large Language Models · entry 01/06

Transformers & attention

The architecture behind modern AI is one idea applied ruthlessly: let each token decide, with learned weights, which other tokens matter.

◈ generating · llms-transformers-and-attention

The problem it solved

Older sequence models read text like a narrow pipe — word by word, squeezing everything so far into one running summary. Long-range meaning drowned: by the time the verb arrives, the subject is a memory. The 2017 transformer discarded the pipe entirely.

Attention in one paragraph

For each token, the model computes what it's looking for (a query), what it can offer (a key), and what it carries (a value) — three learned projections of the same token. Every query is compared against every key; the match scores, softmaxed into weights, decide how much of each token's value flows into the current one. "It" attends hard to "the trophy"; a verb attends to its subject. No rule says so — the weights learned that matching this way lowers loss.

Many heads, many layers

Each layer runs dozens of attention heads in parallel — separate learned lenses that specialize (syntax here, coreference there, position elsewhere), their outputs merged, passed through a small feed-forward network, and handed up. Stack that block fifty-plus times and each token's representation becomes a summary of the entire context as relevant to itself. That all-pairs comparison is also the famous cost: quadratic in sequence length, which is why context windows are an engineering battlefield.

Then it's just next-token prediction

At the top, the model emits a probability for every token in its vocabulary as the next one. Generation is: sample, append, repeat. All apparent planning, reasoning and style lives inside that one repeated act — which is the most counterintuitive fact in the field, and the right amount of unsettling.

Failure mode

Reading attention maps as explanations. They show where information could flow, not why an answer was produced — pretty heatmaps have justified many wrong stories. Interpretability is a research field precisely because the honest answer to "why did it say that" is still, mostly, "300 billion weights voted."