Computer Vision · entry 04/06

Video understanding

Watching is not looking 30 times a second — video models must track identity, motion and cause across frames, under token budgets that make every second expensive.

Why video isn't stacked images

A classifier can name what's in a frame; video asks what's happening — and happening lives between frames. "Picking up" versus "putting down" are identical stills in opposite orders; "chasing" is a relationship among trajectories, not a pattern of pixels. Early video networks bolted time onto image backbones with 3D convolutions and motion streams; the transformer era treats a clip as tokens in one more dimension, with attention free to connect the object released at second nine back to the hand that held it at second three. The mechanics carried over; the budget didn't.

The token famine

Video is brutally over-sampled: an hour at 30fps is a hundred thousand frames of mostly redundant pixels, and naive tokenization would drown any context window before the opening credits end. Every practical system is therefore an economy of attention: sample frames sparsely, compress runs of similar content, spend resolution where motion happens, keep a running memory instead of the raw past. When a video model misses something, the first question is not "is it smart enough" but "was that moment still in the budget" — often the model never saw the frame you're asking about.

What works today

Multimodal models now answer questions about clips, localize moments ("when does she open the box"), read screen recordings, summarize meetings and lectures, and — fed egocentric video — narrate what a wearer is doing, which is quietly foundational for robot learning from human video. Reliability falls with duration and with precision of the question: counting repetitions, exact timestamps, and causal chains over many minutes remain the failure zone, for the same reason VLMs wobble on spatial detail — the evidence is thinner than the fluency.

Failure mode

Assuming the model watched what you watched. Ask about minute forty-two and a sparse sampler may answer from priors — what usually happens in such videos — because the relevant seconds fell between sampled frames, and nothing in the answer will say so. For anything load-bearing, ask systems that cite timestamps, spot-check the cited moments, and treat confident answers about uncited moments as guesses wearing a watch.