The Lab · instrument 06/09 · interactive

Teach an agent

A live reinforcement-learning agent in a world you control: draw walls, move the goal, tune exploration, and watch learned values adapt — or mislead — in real time.

What's real here

This is tabular Q-learning, exactly as the RL section describes it: the agent acts ε-greedily, every step nudges a table of state-action values by temporal difference, and the heat map you're watching is that table. The arrows are the greedy policy — what the agent would do with exploration off. Nothing is scripted; the route it finds is whatever your maze and its own experience produce.

Where to push

Let it converge on an empty grid, then draw a wall across its favorite corridor. For the next several episodes the agent walks confidently into your wall — the values behind that policy were earned in a world that no longer exists, and only fresh, surprising experience repairs them. You have just watched distribution shift from the inside. Now try the exploration slider: pin ε near 1 and the agent never cashes in what it knows; pin it at 0.05 from the start and it often locks onto the first mediocre route it stumbled into, because it stopped looking. The decay schedule you can toggle is the standard compromise, and now you know why it exists.

The mean experiment

After convergence, quietly drag the goal somewhere far from its old home. The entire value landscape is now not just stale but actively wrong — every arrow points toward a reward that isn't there. Watch how long the old gradient keeps dragging the agent to the empty corner, and which cells recover first (the ones it's forced through). This is the tabletop version of why reward and environment changes are handled so carefully in real systems, and why deployed policies get monitored rather than trusted.

Failure mode

Reading the heat map as understanding. The glowing values look like the agent "knows the way," but wall off the goal entirely and it will run its 250-step episodes forever without ever representing that the task is impossible — the warning the page then shows you is the page's own bookkeeping, not a thought the agent had, because no value in the table can say "there is no route." The agent knows numbers about actions, nothing else. Capability without a world model fails silently; keep that picture for the next time a fluent system seems to understand its job.