Computer Vision · entry 01/05

How machines see

An image reaches the model as a million meaningless numbers; vision is the art of re-describing them, layer by layer, until objects fall out — first with convolutions, now with attention too.

The input is humbling

A photo arrives as a grid — height × width × three color channels, each cell a number from 0 to 255. Nothing in that grid says "edge," let alone "cat": move the camera an inch and every number changes while the cat stays put. That gap — between pixel values and stable meaning — is the entire problem of computer vision, and for decades hand-written rules lost to it. What won was the layered re-description trick: don't define a cat, learn a chain of translations that ends in one.

Convolution: one small filter, everywhere

A convolutional layer learns a tiny filter — say 3×3 weights — and slides it across the whole image, writing how strongly each neighborhood matches. One filter learns to fire on vertical edges, another on a patch of orange, another on a right angle. Stack layers and the matches compound: edges into textures, textures into parts, parts into objects. The design smuggles in a powerful assumption, translation equivariance — shift the cat and the features shift with it, so a whisker detector works in any corner of the frame — and the network reuses one small set of weights across every position instead of learning each location separately. That efficiency is why, in 2012, a convolutional network called AlexNet cut the ImageNet error rate from 26% to 15% and started the deep-learning era: vision was the field where scale first paid.

Then attention arrived

Vision Transformers (ViTs) discard the sliding filter: chop the image into 16×16-pixel patches, treat each patch as a token, and let attention decide which patches matter to which — a wing patch attending to a beak patch from across the frame. With small datasets this underperforms (it has to relearn what convolutions assume for free); with internet-scale training data the learned version wins, echoing the field's oldest lesson about built-in assumptions versus scale. Modern systems use either, or hybrids, and the practical differences are smaller than the papers imply.

Features are the product

The classifier at the top is almost an afterthought; the valuable artifact is the stack of features below it. Chop off the final layer and the remaining network turns any image into a vector that means something — the visual cousin of a text embedding — which is why one pretrained backbone feeds a hundred downstream tasks, from medical imaging to shelf audits, with a few thousand labeled examples each.

Failure mode

Assuming the machine sees what you see. Networks — convolutional ones especially — lean on texture and context more than shape: a leopard-print sofa reads as leopard; a faint wash of adversarial noise across every pixel, invisible to you, flips "panda" to "gibbon" with high confidence. Human vision and machine vision agree on the easy cases and diverge exactly where stakes concentrate — the unusual, the occluded, the deliberately hostile. Reason from what the features actually respond to, not from the word "see."