Reinforcement Learning · entry 02/05

Value functions & policy gradients

Learn how good each situation is and act greedily — or adjust behavior directly toward what worked. The two families of RL, why deep learning supercharged both, and why training is famously unstable.

Family one: learn values

Value methods learn a scoreboard. Q-learning's table holds "expected future reward if I take action a in state s," updated by temporal difference every step; behavior is just greed over the table (plus exploration). It's elegant and sample-efficient — every step teaches something, and experience can be replayed from a buffer. The catch: tables need one row per state, and real states (a camera frame, a chess position) are astronomically many. Deep RL's breakout result replaced the table with a network — DQN learned Atari from pixels — and inherited a new fragility: the targets the network learns from are computed from the network itself, so error can feed back into error. Stabilizers (frozen target copies, replay buffers) are load-bearing, not optional.

Family two: adjust the policy

Policy-gradient methods skip the scoreboard and tune behavior directly: run the current policy, then shift probability toward actions that preceded better-than-expected outcomes. This handles continuous actions naturally (motor torques have no table rows) and learns stochastic policies where hedging is optimal. The price is variance: "better than expected" measured over a few noisy episodes is a jittery teacher. Actor-critic architectures marry the families — a critic learns values to judge the actor's choices calmly — and the workhorse algorithms (PPO above all) add one more idea: don't let any update move the policy too far from the one that gathered the data. Trust regions are why modern RL trains at all.

The honest state of practice

Deep RL is powerful and temperamental. Runs differing only in random seed can succeed and fail; hyperparameters that work in one environment faceplant in the next; and evaluation is noisy because the policy changes the data it collects — a feedback loop supervised learning never faces. Practitioners rerun everything across seeds, report distributions rather than best runs, and treat any single learning curve as an anecdote. When it works, it finds behavior no dataset contained; getting it to work is the job.

Failure mode

Judging the family by the demo. Value methods look magical on discrete games and stumble on continuous control; policy gradients handle robots and burn samples games would never forgive. Papers benchmark where their method shines. The selection question is mechanical — discrete or continuous actions, simulator or real experience, sample budget — and choosing by which demo impressed you is how six-month projects restart at month four. The same eval-first discipline applies: define your environment's constraints before shopping for algorithms.