Speech & Audio · entry 02/05

Speech to text

Weakly-supervised training at internet scale made transcription a commodity; word error rate tells you how good, and its fine print tells you where it will quietly ruin you.

How recognition works now

Modern batch transcription is a sequence-to-sequence transformer: encode the audio (as spectrogram frames or learned units), decode text tokens one at a time, exactly like a language model conditioned on sound. (Streaming engines lean on CTC and transducer decoders instead — emitting while still listening is precisely what those architectures buy.) The era's defining move was training scale and looseness: Whisper-style systems learn from hundreds of thousands of hours of audio paired with whatever transcripts the internet already had — subtitles, captions, imperfect ones included. The result is transcription that handles accents and background noise well enough to be a commodity API, with punctuation, casing, and dozens of languages included. A decade of pipeline stages — phoneme dictionaries, acoustic models, language models glued with search — collapsed into one network.

The number that matters, and its fine print

Accuracy is quoted as word error rate: insertions plus deletions plus substitutions, over reference length. Single digits on clean English is normal now. The fine print: WER is an average that spreads error unevenly — proper names, drug names, amounts, and addresses fail at multiples of the headline rate, and those are usually the words your application exists to capture. WER also counts "um" handling, casing, and formatting disputes as errors or non-errors depending on normalization choices, so vendor comparisons need identical scoring. Build a test set from your own audio, score the fields you care about separately, and treat the headline WER as marketing weather.

Design decisions that bite

Streaming or batch: live captions need partial results in hundreds of milliseconds and accept lower accuracy; batch jobs can use bigger models with the whole recording as context. Speaker attribution: transcripts without "who said it" are half a product — diarization is its own problem with its own failure rate. Custom vocabulary: most engines accept a boost list for your product names; wiring it in is the single cheapest accuracy win available. And downstream cleanup — an LLM pass that fixes punctuation and formats numbers — often improves usefulness more than a better acoustic model would.

Failure mode

Hallucinated speech. Trained on subtitle data, recognizers learned what subtitles do during silence and music — emit plausible text like "Thanks for watching!" — and they will produce fluent sentences nobody said, especially in quiet stretches, non-speech audio, or a language mismatch. It is the audio edition of plausible-over-true. Gate transcription with voice-activity detection, check segment confidence, and never feed unreviewed transcripts into anything that acts on "what the customer agreed to."