section 11 · status: live · 6 entries · updated 2026-08-29
Reinforcement Learning
No answer key — an agent acts, the world scores it, and credit flows backward through time. The frame, the two algorithm families, the reward-hacking museum, the RL engine inside every frontier assistant, what to do when all you have is a log, and the honest checklist for when to use any of it at all.
live · tabular Q-learning — the value map lights up, the path straightens
- Learning from consequences3 minno answer key — just a world that scores you
- Value functions & policy gradients2 mintwo ways to turn experience into behavior
- Reward design & specification gaming3 minthe agent optimizes the number, never the intention
- RL for language models3 minthe quiet engine inside every frontier assistant
- Learning from logs3 minyou have data, not a simulator
- When RL wins3 minsuperhuman at Go, struggling with a doorknob
check yourselfAnswer before you open
Trying to recall something teaches it better than re-reading does. Have a go, then open the answer.
An RL agent's score climbs steadily while its behavior looks absurd. What is happening?
Specification gaming: it is maximizing the reward you wrote, which is the only specification it has. Rising return with degenerate behavior is the signature — watch behavior, not just the curve. Reward design & specification gaming →
Why do reasoning models train against math and code rather than essays?
Because those answers can be checked mechanically. A verifiable reward cannot be charmed the way a learned judge can, so the behaviors that actually pass get reinforced. Where verification is impossible, the gameable judge returns. RL for language models →
Name two conditions that should make you reach for RL — and one that should stop you.
Reach: sequential decisions with a reward you would defend under adversarial pressure, plus a simulator or survivable real-world exploration. Stop: if demonstrations exist, imitation is cheaper; if a PID controller works, ship the PID controller. When RL wins →
What does it mean for an agent to learn a 'value' as opposed to a 'policy'?
A value function estimates how good a state or action is, and the policy falls out of picking the best one. A policy method learns the action distribution directly. Value methods are sample-efficient and unstable — the targets are computed from the network itself, so error feeds error; policy methods are steadier and hungrier. Most working systems use both. Value functions & policy gradients →
You have three years of logged decisions and no simulator. Can you learn a better policy?
Sometimes — that is offline RL — but the estimate will flatter you. The value function has no data for actions nobody took, so maximization selects its most optimistic errors. Off-policy evaluation only works to the extent the old policy explored, which is why logging a few percent of randomized decisions, with their probabilities, is the highest-value thing to do before you need any of this. Learning from logs →