Reinforcement Learning · entry 05/06 · 3 min read

Learning from logs

Most real systems can't afford exploration — offline RL learns a better policy from decisions already logged, and its central difficulty is that it cannot check its own good ideas.

The setting almost everyone is actually in

Textbook RL assumes you can act and see what happens. Real systems — a lending policy, a treatment protocol, a recommender — usually can't: exploring means making decisions you know are worse, on people. What they do have is history: years of states, actions taken by the existing policy, and outcomes. Offline RL is the attempt to learn a better policy from that log without ever acting during training. It is also where most industrial demand points, which is awkward: the missing simulator is precisely the item the checklist treats as a reason to reach for something simpler.

Why it is harder than supervised learning on the same data

The trap is specific. A learned value function is asked what an unlogged action would have been worth, and it has no data there — so it extrapolates. The errors need not be biased at all; what makes them dangerous is the maximization on top of them. A policy searching for the highest value it can find will select the model's most optimistic mistakes, every time. The classic result is a policy that looks superb by its own estimate and fails in deployment. The remedies all amount to enforced humility: stay close to the behavior in the log, penalize confidence about unseen actions, or simply imitate the best-performing slices of history rather than optimizing at all.

Evaluating without deploying

The other half of the problem is that you cannot score a new policy by running it — that was the constraint in the first place. Off-policy evaluation estimates how a proposed policy would have done using logged data, and it is trustworthy in proportion to how much the old policy explored. Logs from a deterministic system support almost no counterfactual, which is why the highest-leverage thing a team can do before wanting offline RL is to log a little deliberate randomization — a few percent of decisions made differently, recorded with their probabilities. That single habit turns a log into a dataset.

Failure mode

Trusting the offline number. Every estimator has a way of lying to you here: a model-based one shares the value function's own blind spots, and an importance-weighted one is unbiased but wildly high-variance exactly where the log's coverage is thin. Then you compare a dozen candidates and keep the best-scoring, so the winner is partly whichever one got the luckiest estimate. Treat the number as a screening filter: reject bad candidates cheaply, run the survivor in shadow mode to see how far and how often it diverges from the incumbent, then let a small canary tell you what those divergent actions are actually worth — because that outcome is the one thing your log never contained.