History of AI · entry 04/05

The deep-learning decade

2012–2022: vision falls, attention is invented, scale becomes a strategy, and a chat interface turns a research curve into a public event.

Conquest by benchmark

After AlexNet, the pattern repeated on a schedule: pick a field, apply deep networks and scale, watch the previous tradition retire. Speech recognition error rates collapsed. Machine translation went end-to-end with seq2seq (2014). Go — supposedly a decade away — fell to AlphaGo in 2016, with self-play then rederiving the game from scratch in AlphaZero. AlphaFold (2020) effectively solved protein structure prediction, deep learning's clearest gift to science so far. Each conquest recruited more talent and more compute for the next.

The architecture and the laws

2017's "Attention Is All You Need" replaced recurrence with attention and, mostly unnoticed at the time, provided the substrate everything since runs on. BERT and GPT (2018) added the recipe: self-supervised pretraining on raw text, then adaptation. GPT-2 and GPT-3 (2019–2020) demonstrated the strange economics — smooth, predictable loss curves hiding jumpy capability arrivals like few-shot learning that nobody explicitly trained. (GPT-2's staged, worried release in 2019 — capability judged too risky to drop all at once — now reads as the dress rehearsal for every frontier-release debate since.) Scale stopped being a budget line and became the strategy itself, formalized by scaling-law papers that let labs purchase capability on a forecast. Sutton's "bitter lesson" (2019) named the pattern: general methods riding compute beat human-crafted knowledge, every time, eventually.

The deployment event

ChatGPT (November 2022) contained modest research novelty — GPT-3.5 plus RLHF and a chat box — and produced the fastest consumer-software adoption curve anyone had yet seen: a hundred million users in two months. The lesson is uncomfortable for researchers and vital for everyone else: the capability had existed for years; the interface made it legible. The decade ended with AI as a consumer product, a boardroom agenda, and a regulatory subject — states of being research alone never achieves.

Failure mode

Reading the decade as inevitability. Survivor bias flattens it: for every bet that compounded, rivals died quietly — capsule networks, most of symbolic revival, a hundred architectures. The compounding was real but the path was contingent, funded by gaming hardware and one benchmark's credibility. Extrapolating "scale always wins" as a law of nature — rather than a strategy that has kept winning while its inputs kept getting cheaper — is exactly the kind of confident induction this field's history exists to discipline.