Generative Media · entry 01/04

Diffusion models

How iterative denoising, latent spaces, and cross-attention turned random static into the engine behind modern image generation.

Learn to remove a little noise

Every diffusion model is trained on one deceptively small task. Take a real image, add a known amount of random noise, and have the network estimate that noise so it can be subtracted. Sometimes the image is barely dusted, sometimes it is nearly pure static, and the loss is plain regression on the estimate — no adversary, none of the instability that made GANs miserable to train. The mechanics are ordinary gradient descent; the insight is the task. A network that can clean up noise at every severity has implicitly learned what images look like.

Generation runs the tape backward

To generate, start from pure static — a canvas with no image in it at all — and apply the denoiser in a loop. Each step removes a slice of noise and commits to a little more structure: rough masses first, edges and texture late. Prompt guidance folds into the same loop: the model predicts the noise twice, with and without the text, and the sampler exaggerates the difference.

x = noise()                      # pure static
for t in schedule:               # ~20-50 steps, coarse to fine
    e_c = model(x, t, prompt)    # noise estimate, conditioned
    e_u = model(x, t, empty)     # noise estimate, unconditioned
    e = e_u + g * (e_c - e_u)    # g = guidance scale
    x = step(x, e, t)            # subtract one slice of noise
image = decode(x)                # latents -> pixels

The guidance scale g is a literalness dial, not a quality dial: low values drift loosely around the prompt, moderate values follow it, extreme values overshoot.

How the prompt gets in

Text conditioning works through cross-attention. A frozen text encoder turns the prompt into a sequence of embeddings, and at every denoising step, layers inside the denoiser attend over those vectors — the same attention mechanism that powers language models, pointed at an image in progress. Regions of the canvas learn to look at the tokens that describe them, which is why word choice shapes composition, not just content.

Latent space pays the bills

Running dozens of denoising steps over a megapixel of raw pixels would be brutally expensive. Latent diffusion is why any of this is affordable: a separately trained autoencoder compresses the image about 8x per side, the entire denoising loop runs in that small latent space, and one decoder pass at the end produces pixels. The efficiency frontier has since moved to the step count itself. Flow-matching and consistency-style models learn straighter, more direct paths from noise to image, collapsing fifty steps toward four, two, even one — the difference between a render farm and a phone.

Failure mode

Cranking guidance to fix a bad image. When outputs disappoint, the reflex is to raise the scale, and the results come back on-prompt but fried: blown saturation, waxy skin, every element shouting for attention. High guidance also collapses diversity, so ten samples converge on one idea. When a generation is off, change the seed, the step count, or the wording first; move guidance last, and in small steps.