Reinforcement Learning · entry 04/05

RL for language models

RLHF is reinforcement learning with a learned judge; reasoning models train against checkable answers. How RL went from games to the center of the LLM pipeline — and what it amplifies.

The mapping

Take the RL frame and substitute: the policy is the language model, an action is emitting a token, an episode is a full response, and the reward arrives at the end — from a reward model trained on human preferences. That's RLHF: policy-gradient training (PPO and successors) against a learned judge, with one crucial leash — a KL penalty that punishes drifting far from the reference model it started from (the supervised fine-tune, not the raw base), so the policy improves without collapsing into reward-hacking gibberish. Simpler variants (DPO and kin) fold the same preference signal into a supervised-style loss and skip the full RL machinery; the industry uses both, but the frame — optimize a proxy judge, under a leash — is the same.

Verifiable rewards changed the game

A learned judge can be flattered; every reward model is a gameable proxy. So the sharpest recent progress came where judges are incorruptible: math with checkable answers, code with unit tests, puzzles with verifiers. Sample many attempts, reward the ones that actually pass, reinforce the reasoning that produced them — RL with verifiable rewards is the engine behind reasoning models, teaching longer chains of thought, backtracking, and self-checking because those behaviors won under a judge that could not be charmed. The boundary of the method is the boundary of verification: where correctness can't be checked mechanically, the gameable judge returns.

What RL does and doesn't add

Pretraining installs most of the capability; RL mostly redistributes probability over it — sharpening behaviors that earn reward, suppressing ones that don't — and how much wholly new skill long RL runs add on top is genuinely contested. Preference tuning, at least, works with astonishingly little data compared to pretraining, and its pathologies are amplifications: sycophancy is agreement rewarded a hair too often; over-refusal is caution overpaid; confident wrongness is what a hurried rater couldn't catch. Every quirk of a frontier assistant is downstream of some judge's blind spot — which is the alignment stack's standing challenge, restated.

Failure mode

Forgetting the judge is in the loop forever. Teams treat the reward model as ground truth and scale optimization against it — and optimization against a frozen proxy eventually detaches it from what you wanted (over-optimization is measurable: real quality rises, then falls, while proxy reward climbs throughout). The defenses are unglamorous: keep the KL leash on, refresh the judge with new human data, hold out evals the judge never sees, and read actual transcripts. The moment your only quality signal is the number you're optimizing, you are the boat spinning through checkpoints.