Speech & Audio · entry 01/05

How machines hear

A microphone delivers thousands of numbers per second with no words in them anywhere — hearing starts by repainting that stream as a spectrogram, and then treating listening like seeing.

The raw material

Audio arrives as a waveform: air pressure sampled thousands of times per second — 16,000 for speech systems, 44,100 for music — each sample one number. A second of speech is sixteen thousand numbers in which nothing resembles a phoneme, a word, or a speaker; the same sentence said twice never produces the same numbers. Like pixels, samples are the wrong level of description, and the first job is re-description.

The spectrogram move

The standard trick: slice the waveform into overlapping frames a few dozen milliseconds long, measure how much energy each frame carries at each frequency, and stack the slices left to right. The result is a spectrogram — time across, frequency up, brightness for energy — and it makes speech visible: vowels appear as stacked harmonic bands, consonants as bursts and hisses, intonation as a drifting fundamental. Squash the frequency axis onto the mel scale (fine resolution where human ears have it, coarse where they don't) and you have the input that fed a decade of speech systems. The deep insight is almost a joke: once sound is a picture, the whole vision toolkit applies — convolutions over local patterns, and lately attention across the whole utterance.

Learning to hear without labels

Transcribed speech is scarce and expensive; raw audio is infinite. Many modern audio models therefore start self-supervised: mask a slice of the audio and train the model to predict it from context, the listening version of fill-in-the-blank pretraining. Models like wav2vec and its descendants learn phoneme-like units, speaker characteristics, and acoustic structure from thousands of hours of unlabeled sound — then a little labeled data fine-tunes that foundation into a transcriber, a speaker identifier, or a sound-event detector. Some systems now skip the spectrogram entirely and learn their own front end from raw waveform; the pipeline is a convenience, not a law.

What's hard is everything around the words

One recording rarely contains one clean voice. Rooms echo, cafés murmur, two people overlap, the microphone is cheap and three meters away. Separating sources, canceling echo, deciding who is speaking when — these front-end problems consume more engineering than recognition itself, and they're why the demo that worked on a headset mic dies in a conference room.

Failure mode

Testing on clean audio. Every audio benchmark skews toward studio-quality recordings, and accuracy measured there survives first contact with a speakerphone about as well as a vision model survives rain. The distribution gap in audio is brutal — accent, distance, noise, compression — and it is the gap, not the model, that decides your production accuracy. Collect real audio from your real channel before believing any number.