The Lab · instrument 02/09 · interactive

Walk the loss landscape

Gradient descent with your hand on the two knobs that matter — learning rate and momentum — on a landscape with real local minima to get stuck in.

What's real here

The update rule is the one from how models learn, verbatim: measure the slope where the ball stands, step downhill, repeat. Momentum, when you raise it, blends in the previous step's direction — a rolling memory that smooths the ride. The landscape is one weight's slice of a loss surface; training a real model is this exact walk in billions of dimensions at once.

Where to push

Drop three balls in different places with the default settings and watch them find different valleys — where you start decides where you end, which is why initialization is a research topic and not a formality. Now raise the learning rate past 0.5 and drop a ball near a steep wall: each step overshoots the valley floor, the overshoot steepens the next gradient, and the ball rockets out of the plot. That's divergence, and the loss curve of a real run doing this is a vertical line — every practitioner has one framed somewhere. Then set the rate to 0.005 and appreciate the opposite failure: perfect stability, geological progress.

Momentum earns its keep

Find a run where a ball parks in a shallow dip short of a deeper valley. Clear it, raise momentum to 0.9, drop the ball in the same place: it carries speed through the dip and settles somewhere better. The same trick — variants of it power the optimizers behind every frontier training run — buys progress through flat, boring stretches of landscape that pure gradient following would crawl across.

Failure mode

Taking the one-dimensional intuition upstairs. In this plot, local minima are the villain; in billions of dimensions, true traps are rare — to be stuck you must be stuck in every direction at once — and the practical enemies are saddle points and vast flat plateaus. The picture teaches the mechanics honestly and the geometry misleadingly. Keep the update rule, upgrade the landscape.