Generative Media · entry 04/05

3D & world models

Radiance fields and Gaussian splats made 3D capture cheap; world models generate interactive environments frame by frame — two roads from images toward space you can move through.

Capture got solved first

The quiet revolution: turning photos of a thing into the thing. Neural radiance fields (NeRFs) learned a scene as a function — query a point and direction, get color and density — reconstructing photoreal 3D from ordinary snapshots at the cost of slow rendering. Gaussian splatting replaced the network with millions of oriented, translucent blobs that rasterize in real time, and 3D capture jumped from research demo to product feature: scan a room with a phone, walk a buyer through it tonight. This is reconstruction, not imagination — its raw material is multi-view geometry plus optimization, and its failure cases (reflections, thin structures, anything that moved) are geometry's failure cases.

Generation followed

Text-to-3D rides on image models: use a 2D diffusion model as a critic scoring renders from many angles, and optimize a 3D representation until every view satisfies it — or, increasingly, generate multi-view images and reconstruct directly. Assets that took an artist days now take minutes, at "excellent draft" quality: game studios and product teams generate, then retopologize and polish by hand. The craft transferred intact from image practice: iterate wide, select hard, fix in post.

World models: space as video

The stranger frontier skips geometry entirely. A world model predicts the next frame given a short window of recent frames and your actions — hold it in a loop and you get an explorable environment generated on the fly: playable game worlds dreamed at interactive rates, driving scenarios that respond to steering, neural simulators training robot policies on experience no one had to stage. Consistency is the open wound — walk a corner and back, and the room may have quietly redecorated, because nothing persists except what the model happens to keep "in mind." The fix — memory, geometry anchors, longer contexts — is the same fight video generation is having, with a joystick attached.

Failure mode

Confusing the two roads. Teams pitch splats when they need imagination (a scan can't invent the unbuilt penthouse) or world models when they need measurement (a dreamed room has no reliable dimensions — nothing guarantees the hallway is 1.2 meters wide, or the same width twice). Reconstruction answers "what is there"; generation answers "what could be there"; and any workflow where money follows the measurements should be suspicious of pixels that were never observed.