Speech & Audio · entry 04/05

The voice loop

A voice agent is a pipeline of hear-think-speak stages racing a conversational clock, or one native audio model — the tradeoff between the two is the current frontier.

The pipeline

The standard voice agent chains four stages: voice-activity detection decides someone is speaking, speech-to-text transcribes it, a language model — usually an agent with tools — decides what to do, and synthesis answers. Each stage works on its own; the product is the plumbing between them. Humans notice a gap in conversation at around 200 milliseconds and judge the exchange dead past a second, so the whole chain runs streamed and overlapped: transcribe while they speak, start the model early, begin speaking the first clause while the rest is still being generated.

Barge-in, the make-or-break feature

Real people interrupt. A usable system keeps listening while it talks, detects the interruption, stops its own mouth, and — the subtle part — revises its record of what it actually got to say, because the user is responding to the words they heard, not the paragraph you queued. Systems without honest barge-in produce the signature failure of phone bots: two parties talking over each other, neither updating. Getting this right is echo cancellation, endpoint detection, and context management braided together.

Native speech-to-speech

The newer construction skips text: one model consumes audio tokens and emits audio tokens, trained end-to-end on conversation. What that buys is everything transcripts destroy — it hears tone, hesitation, sarcasm, and it can laugh, pause, or match energy in its reply, with latency one stage instead of four. What it costs: the reasoning transplant is incomplete (the strongest thinking still lives in text-first models), tool use and audit trails want text anyway, and you give up some of the pipeline's inspectability — any transcript you log is a reconstruction, not the representation the model actually conditioned on. Many products land on a hybrid: native models for the conversational surface, text agents for the heavy lifting behind it.

Failure mode

Optimizing intelligence while the clock loses the user. Teams polish the model's answers while the pipeline sits at two seconds of round-trip, and every test caller hangs up before hearing the genius. In voice, latency is a feature users can measure with their skin; past the first second, a smarter answer is a slower failure. Instrument the full loop — mic to first audible syllable — and spend on the slowest stage, which is almost never the one your team enjoys improving.