Large Language Models · entry 03/06

How LLMs are trained

A frontier model is built in stages — pretraining buys knowledge, fine-tuning buys behavior, preference optimization buys judgment — and each stage exists to fix the previous one's failure.

Stage 1: pretraining

Predict the next token, trillions of times, across a filtered slice of the internet, books and code. Months of thousands-of-GPU time and nearly all the cost live here. The result is a base model: a vast completion engine with startling knowledge and no manners — ask it a question and it may answer, continue your question with three more, or write a forum thread where someone eventually replies. It models text, not conversation.

Stage 2: supervised fine-tuning

Show it tens of thousands of exemplary dialogues — question in, good answer out — and train briefly on those. The model learns the format of being an assistant: answer the question, use the person's language, stop when done. Cheap relative to pretraining, decisive for usability. Skill ceiling: it imitates the demonstrations; it can't exceed them.

Stage 3: preference optimization

Generate multiple answers, have humans (increasingly: humans assisted by AI, or an AI judging against a written constitution) pick the better one, and optimize toward preferred answers — RLHF and its successors. This is where helpfulness, tone, refusals and judgment get shaped: not "what does a good answer look like" but "which of these is better." It's also where distortions enter: reward models that overvalue confidence, length, or agreeableness train exactly those.

The pipeline explains the quirks

Sycophancy, hedging, over-refusal, confident error — each traces to a stage: the internet's habits from pretraining, demonstrators' style from SFT, the reward model's blind spots from preference training. When a model's behavior puzzles you, ask which stage made that behavior win.

Failure mode

Believing alignment is a coat of paint. The base model's capabilities and biases remain underneath the assistant persona; unusual prompts can reach around the trained behavior. The stages bias the distribution — strongly, usefully — but nothing was deleted, which is why safety work is ongoing rather than solved.