Computer Vision · entry 03/05

Vision-language models

Contrastive training taught images and text to share a vector space; bolting a vision encoder onto an LLM taught models to talk about what they see. Both tricks changed what's buildable.

The contrastive trick

CLIP-style training takes hundreds of millions of image–caption pairs from the web and trains two encoders — one for images, one for text — with a single objective: a real pair's vectors should sit close; a shuffled pair's far apart. No labels, no taxonomy, just supervision scavenged from data that already existed. The result is a shared embedding space where "a photo of a rusty bicycle" lands near photos of rusty bicycles. Classification stops requiring training: embed your labels as sentences, embed the image, pick the nearest — zero-shot recognition of categories the model was never explicitly taught. Image search, deduplication and content moderation got the same upgrade for free.

Models that talk about pictures

The second construction feeds vision into a language model: an image encoder turns the picture into a sequence of vectors, a small adapter translates them into the LLM's context, and from the model's point of view the image is just unusual tokens to condition on. Now it answers questions about the photo, reads the chart, explains the meme, drafts the alt text. This is what "multimodal" means in practice — and it's why agents can operate software from screenshots: the screen is one more image to reason over.

Where the seams show

VLMs inherit language-model habits and add visual ones. Counting beyond a handful is unreliable; fine spatial relations ("left of," "behind") wobble; small text may be below the encoder's resolution — many pipelines quietly downscale your 4K screenshot to something far coarser, and the model never saw what you think you showed it. Know your system's input resolution; crop and zoom before asking about detail.

Failure mode

Visual hallucination — answering from priors instead of pixels. Ask about a detail that isn't clearly visible and the model reports what such scenes usually contain: a watch reads 10:10, the license plate is invented, the brand on the blurry label is confidently named. It's the same plausible-continuation objective, now with an alibi ("I looked"). Grounding helps — ask it to quote the exact region, request "not visible" as an allowed answer, verify anything load-bearing with a crop — but design as if sight, like fluency, can be faked.