Large Language Models · entry 09/11 · 3 min read

One model, many senses

Frontier models stopped being text engines with an image adapter and started training on everything at once — what that actually changes, and what it costs.

Two ways to build a multimodal model

The first generation grafted senses on. Take a finished language model, train a separate image encoder, and learn a small adapter that projects its output into the language model's token sequence. The model then attends over text and image positions together — cross-modal reasoning is available, and that is the point of the design. The limits are elsewhere: the encoder was optimized for a different objective, the adapter is a narrow channel between the two, and the language model's own pretraining never required it to ground anything in pixels. The second generation trains on interleaved text, images, audio and video from the start. There is still a separate front end per modality — a patch embedder for images, a spectrogram stage for audio — but they feed one stream through one transformer trained end to end, so the representation is shared rather than translated. Most shipping models sit somewhere between the two.

What the shared representation buys

Mostly efficiency and reach. Controlled comparisons at matched compute have not found an inherent capability advantage for either design; what training jointly from the start buys is the same quality with fewer parameters and less training cost, which is why the frontier moved. What it plausibly also buys — and what the grafted pipelines are worst at — are the tasks living between modalities: reasoning about a chart while reading the paragraph that discusses it, hearing tone and reading words and noticing they disagree, watching a clip and tracking what changed. And it buys output: a model that generates images in the same loop it understands them can edit conversationally, which is a consistency advantage over re-prompting a separate generator, if not always a fidelity one.

What it costs

Everything gets more expensive and more entangled. Images cost hundreds of tokens each and video costs them every second, so a minute of video runs to a long report and an hour fills a million-token window — audio, by contrast, is comparatively cheap. That makes context budget the binding constraint on any real video task. Training data must be interleaved and aligned, which is far scarcer than text. Evaluation multiplies, because a model can be excellent on images and quietly poor on audio — currently the weakest modality across the board — and one headline benchmark hides that entirely. And the attack surface widens: an instruction can be hidden in an image or a waveform, which is the same injection problem arriving through a channel your text filters never see.

Failure mode

Assuming the weakest modality is as good as the strongest. Marketing speaks of one model, so teams test on text, find it excellent, and ship a feature that depends on reading a photographed receipt or a noisy phone recording. Perception degrades in exactly the ways the vision entry and the speech entry describe — resolution, lighting, background, accent, noise — and a shared architecture repeals none of it. Build a small eval per modality, on your own inputs, before you believe the single number.