AI Infrastructure & MLOps · entry 01/05

The hardware layer

Modern AI is shaped by silicon economics: why GPUs won, why memory bandwidth is the real bottleneck, and why the interconnect bill rivals the compute bill.

Why GPUs won

A neural network is matrix multiplication in a trench coat, and matrix multiplication is thousands of independent multiply-adds — exactly the workload graphics cards were built for. Where a CPU spends its silicon on a few sophisticated cores that handle anything, a GPU spends it on thousands of simple ones that handle the same operation on different data. Deep learning's arrival was less an invention than a discovery that the gaming industry had spent twenty years accidentally building its substrate. Purpose-built accelerators (TPUs and kin) push the same bet further: silicon organized around the tensor operations that dominate the workload, general-purpose everything else be damned.

The memory wall

The unintuitive truth: compute is rarely the constraint — feeding it is. A GPU can multiply far faster than memory can deliver operands, so utilization lives and dies on arithmetic intensity: how much math you do per byte fetched. This is why high-bandwidth memory (HBM) stacked beside the die is the component the industry actually fights over, why token-by-token decoding is bandwidth-bound (streaming billions of weights per token), and why precision keeps dropping — FP32 gave way to BF16, FP8, even 4-bit weights, each halving the bytes per operation at a small, usually survivable, accuracy cost. Half the software tricks in this section are memory tricks wearing a math costume.

The interconnect is the computer

Frontier models don't fit on one chip, so chips are lashed together — high-speed links within a server, specialized fabrics across racks — and suddenly the network is part of the arithmetic: a distributed training step is compute punctuated by synchronized gradient exchanges, and a slow link idles ten thousand expensive chips in lockstep. Datacenter AI design is therefore topology design: what talks to what, at what bandwidth, with what failure domains — plus the unglamorous constraints that actually gate buildouts now: megawatts, cooling, and land next to power.

The supply chain is strategy

Advanced accelerators depend on a handful of chokepoints — leading- edge fabs, HBM production, advanced packaging — concentrated in very few companies and geographies. That concentration is why compute became a lever of geopolitics, why export controls target chips and the tools that make them, and why every major lab's roadmap includes some hedge: custom silicon, multi-vendor stacks, or long-term capacity contracts signed years out.

Failure mode

Budgeting FLOPs and ignoring everything that feeds them. Teams size clusters by peak compute, then discover utilization at 30% — starved by data loaders, stalled on network collectives, thrashing on memory. The honest metric is goodput: useful training progress per dollar of wall-clock, measured end to end. A profiler session that finds the actual bottleneck routinely outperforms a hardware upgrade that assumed it.