Reinforcement Learning · entry 05/05

When RL wins

Self-play conquered games, RL now tunes datacenters and chips and reasoning models — and yet most control problems still shouldn't use it. The track record, and the checklist.

The trophies are real

Backgammon yielded first — TD-Gammon learned near-champion play through the 1990s by TD-learning and self-play; Atari fell to DQN from raw pixels; Go — a decade "away" for four decades — fell to AlphaGo, and then AlphaZero rederived opening theory from nothing but the rules, finding moves professionals now study. The common ingredients: a perfect simulator (the game itself), abundant experience, and a checkable score. Where all three held pure — backgammon, chess, Go — self-play added its beautiful trick, an opponent always exactly at your level, and RL didn't just match humans; it left the human tradition entirely. StarCraft and Dota fell too, but needed help — human replays to bootstrap, shaped rewards to get moving: the seams show once the game stops being clean.

The quiet industrial wins

Outside games, RL earns its keep where decisions are sequential, feedback is measurable, and a simulator or fast real loop exists: building and datacenter cooling (roughly ten-percent energy cuts in the published RL trials), chip floorplanning (deployed, and disputed), plasma shape control in fusion reactors, inventory and bidding systems — and, structurally everywhere now, the training of language models. The pattern holds: a well-instrumented environment, a defensible reward, and consequences cheap enough to explore. Note what's absent from the list: open-ended physical manipulation — the real world is a simulator nobody debugged, and robotics wrestles it on its own terms.

The costs, stated before you start

RL is sample-hungry (millions of episodes is normal), unstable across seeds, and merciless about reward mistakes. Sim-to-real gaps eat naive transfer. And the baseline question is mandatory: if decent behavior can be demonstrated, imitation learning is cheaper; if the problem is one decision rather than a sequence, it's a bandit or plain optimization; if a PID controller works, ship the PID controller. The winning deployments mostly use RL as the last mile — polish on top of demonstrations, physics, or a pretrained model — not as the whole vehicle.

Failure mode

Choosing RL for the narrative. "The system learns from experience" demos beautifully and funds readily, and eighteen months later the team has rediscovered that their problem was a lookup table wearing a cape. The checklist is short: sequential decisions? measurable reward you'd defend under adversarial pressure? a simulator or survivable real-world exploration? a budget for instability? Four yeses, use RL — it will find things nothing else can. Fewer, use the boring thing first; the boring thing usually wins, and it never spins the boat.