Machine Learning Foundations · entry 04/06

Where data comes from

Web crawls, labeling workforces, licensing deals, and synthetic generators — every model is downstream of a data supply chain, and most model behavior is explained there.

The supply chain nobody demos

Model launches show benchmarks; they rarely show the pipeline that mattered more. Frontier pretraining data starts as web crawls — billions of pages, most of them garbage — plus books, code, and papers, then passes through the deciding stage: filtering. Deduplication (near-copies teach nothing and amplify memorization), quality classifiers choosing which slices survive, language and toxicity filters, contamination sweeps hunting benchmark answers. Two labs crawling the same web produce different models because they filter differently; curation is a competitive secret precisely because it works.

The human layer

Above the crawl sits paid judgment. Supervised examples and preference ratings come from a real workforce — thousands of contractors grading outputs against style guides, escalating edge cases, drifting in their agreement — and every one of their habits becomes a model habit. The signal's origin is the first place to look when behavior puzzles you: a model that hedges learned hedging from someone's rubric. Specialist data is the expensive frontier — expert demonstrations of medical reasoning or agentic tool use cost orders of magnitude more per example than casual labels, which is why labs increasingly pay professionals, not crowds.

Synthetic data, carefully

Models now generate much of their successors' training data: distillation (a strong model teaches a small one), self-play over verifiable problems, rephrasings that multiply scarce examples. It works — some celebrated small models are mostly synthetic-fed — with one boundary and one hazard. The boundary: synthetic data transfers and reshapes capability; wholly new knowledge still has to enter from outside. The hazard: recursive training on model output can narrow distributions generation by generation — tails vanish, quirks compound — so serious pipelines anchor on human data, verify synthetic examples against checkable ground truth, and measure diversity instead of assuming it.

Data as strategy

Scale made data the scarce input. The open web is largely crawled; publishers now license or litigate (the courts are redrawing the rules); platform owners sit on conversational gold; and proprietary interaction data — what users accepted, corrected, abandoned — is the moat copycats can't crawl. When you evaluate a model claim, ask the supply-chain question first: what did it eat, who chose that, and what would its maker pay to eat next?

Failure mode

Blaming architecture for a data problem. Teams see bias, gaps, or weird verbal tics and reach for bigger models or cleverer prompts, when the cause is upstream: a filter that deleted a dialect, a labeling guide with an opinion, a crawl date that ended the world in 2024. The model is the dataset, rearranged — measure the dataset's failures before buying compute for them.