AI Infrastructure & MLOps · entry 01/09 · 3 min read

The hardware layer

Modern AI is shaped by silicon economics: why GPUs won, why memory bandwidth is the real bottleneck, and why the interconnect bill rivals the compute bill.

Why GPUs won

A neural network is matrix multiplication in a trench coat, and matrix multiplication is thousands of independent multiply-adds — exactly the workload graphics cards were built for. Where a CPU spends its silicon on a few sophisticated cores that handle anything, a GPU spends it on thousands of simple ones that handle the same operation on different data. Deep learning's arrival was less an invention than a discovery that the gaming industry had spent twenty years accidentally building its substrate. Purpose-built accelerators (TPUs and kin) push the same bet further: silicon organized around the tensor operations that dominate the workload, general-purpose everything else be damned.

The memory wall

The unintuitive truth: compute is rarely the constraint — feeding it is. A GPU can multiply far faster than memory can deliver operands, so utilization lives and dies on arithmetic intensity: how much math you do per byte fetched. This is why high-bandwidth memory (HBM) stacked beside the die is the component the industry actually fights over, why token-by-token decoding is bandwidth-bound (streaming billions of weights per token), and why precision keeps dropping — FP32 gave way to BF16, FP8, even 4-bit weights, each halving the bytes per operation at a small, usually survivable, accuracy cost. Half the software tricks in this section are memory tricks first and arithmetic second.

The interconnect is the computer

Frontier models don't fit on one chip, so chips are lashed together — high-speed links within a server, specialized fabrics across racks — and suddenly the network is part of the arithmetic: a distributed training step is compute punctuated by synchronized gradient exchanges, and a slow link idles ten thousand expensive chips in lockstep. Datacenter AI design is therefore topology design: what talks to what, at what bandwidth, with what failure domains — plus the unglamorous constraints that actually gate buildouts now: megawatts, cooling, and land next to power.

The supply chain is strategy

Advanced accelerators depend on a handful of chokepoints — leading- edge fabs, HBM production, advanced packaging — concentrated in very few companies and geographies. That concentration is why compute became a lever of geopolitics, why export controls target chips and the tools that make them, and why every major lab's roadmap includes some hedge: custom silicon, multi-vendor stacks, or long-term capacity contracts signed years out.

Failure mode

Budgeting FLOPs and ignoring everything that feeds them. Teams size clusters by peak compute, then discover utilization at 30% — starved by data loaders, stalled on network collectives, thrashing on memory. The honest metric is goodput: useful training progress per dollar of wall-clock, measured end to end. A profiler session that finds the actual bottleneck routinely outperforms a hardware upgrade that assumed it.