Reinforcement Learning · entry 01/05

Learning from consequences

Reinforcement learning drops the labeled dataset entirely: an agent acts, the world responds, and learning means assigning credit across time to the choices that mattered.

A different contract with the data

Supervised learning gets the answer with every example. Reinforcement learning gets a number, occasionally: the agent observes a state, picks an action, the environment returns a new state and a reward — often zero for hundreds of steps, then suddenly +1 for a win nobody explains. Chess offers no label for move 23; it offers checkmate, forty moves later. The formalism (a Markov decision process) is just this loop written down: states, actions, transition dynamics, rewards, and a goal of maximizing reward summed over time — with future reward discounted, because a payoff now is worth more than a maybe later.

Credit assignment, the core puzzle

When the win arrives, which of the two hundred choices earned it? That's credit assignment, and it is the field's defining problem — backpropagation splits blame through a network's arithmetic, but here blame must also flow backward through time, across actions whose consequences arrived late. The founding trick is temporal-difference learning: maintain a running estimate of each situation's value, and when a moment turns out better or worse than estimated, nudge the estimates of the moments that led there. Predictions teach predictions; the win propagates backward one bootstrap at a time.

Explore, or exploit

A supervised model never faces this: its data exists whether or not it acts. An RL agent only learns about actions it tries, so it must spend some of its life acting suboptimally on purpose — taking the weird route to find out — while still cashing in what it knows. Too little exploration and it locks onto the first decent habit forever; too much and it never benefits from learning. Every practical system schedules the tradeoff (act randomly with probability ε, decaying over time, is the classic), and the tension never fully resolves because it can't: information has a price paid in reward.

Why this frame keeps mattering

RL is the natural language for anything that acts — games, robots, traffic lights, trading, dialogue — and lately for shaping the behavior of language models themselves, where "the environment" is a conversation and "reward" is a judgment of the outcome. Whenever you hear "the system learned by trying things," this loop is underneath.

Failure mode

Assuming the agent learned what you'd have learned. An RL agent learns exactly what produced reward under its exploration, in its environment — including exploits, superstitions, and shortcuts no human would consider the point. The win condition you wrote is the whole specification; the agent read nothing else. Before trusting a learned policy, watch it act in situations it didn't train on — that's where you find out what it actually learned, and reward design is where the trouble begins.