Machine Learning Foundations · entry 01/05

What a neural network is

Strip away the brain metaphors and a neural network is a stack of adjustable arithmetic — which is exactly why it works.

The unit is embarrassingly simple

One "neuron" multiplies each of its inputs by a weight, adds the results plus a bias, and passes the sum through a small nonlinear function. That's the whole unit: output = f(w1*x1 + w2*x2 + ... + b). The weights are the knobs; everything a network "knows" is stored in them.

Why the bend matters

Without the nonlinearity, stacking layers would be pointless — a chain of weighted sums collapses into one weighted sum, and the network could only ever draw straight lines through its data. The bend (modern networks mostly use ReLU-family functions: zero below a threshold, linear above) lets stacked layers compose simple bends into arbitrarily intricate shapes. Depth is a manufacturing process for complexity out of trivial parts.

Layers as re-description

Each layer re-describes its input in terms the next layer finds more useful. In an image model, early layers respond to edges, middle layers to textures and parts, late layers to whole objects. Nobody programs that hierarchy — it emerges because it's the most efficient way to be right, and the network is relentlessly pushed toward being right.

Scale, honestly stated

A large language model is this same unit, repeated. The differences that matter are architectural wiring (how layers connect — see attention) and count: billions to trillions of weights instead of dozens. Nothing qualitatively new is added at scale except the things that, surprisingly, emerge from it.

Failure mode

Taking the neuron metaphor seriously. Real neurons spike, adapt, and wire chemically; artificial ones are rows in a matrix multiplication. The metaphor built the field's vocabulary, but reasoning from biology to model behavior produces confident nonsense — reason from the arithmetic instead.