How do AI voice agents work?

In detail

The chained pipeline

The older design joins separate components. The Moshi paper lists the usual chain: voice activity detection to notice when someone is speaking, speech recognition to turn it into text, a text dialogue model to decide the reply, and text-to-speech to say it. Its authors note that this chain can add several seconds of delay between turns and loses information that is not in the words, such as tone.

Each handoff also adds a chance to be wrong. Smartling’s Olga Beregovaya describes the speech-translation version of this chain: recognize the speech, translate it, render the translation, then generate speech. Every step you introduce adds another potential error. She says a multilingual, multimodal model can now fit that whole chain into one model.

The pipeline has strengths. After speech recognition and before speech synthesis, text checkpoints let you log, test and filter the transcript and the model’s reply before anything is spoken, and you can swap one component without touching the others. Superhuman’s Loïc Houssier says that turning speech into text is the easy part of his team’s voice feature; it can even run on the device. The hard part is turning a rambling dictation into a well-written email, which is the language model’s job.

Speech-to-speech models

The newer design uses one multimodal model that hears and speaks. Liquid AI’s Maxime Labonne describes his company’s audio foundation model as handling any combination: speech to text, text to speech, speech to speech or text to text.

Hosted APIs package this for developers. OpenAI’s Realtime API guide says its model works directly with audio, keeps the conversation state and can call tools, and recommends it for voice agents that need barge-in, low delay before the first audio, natural turn-taking and tool use during the conversation. The session runs over WebRTC in a browser or a WebSocket from a server. Google’s Gemini Live API describes low-latency voice and vision sessions over a stateful WebSocket, with barge-in and function calling. (Both checked October 24, 2026.)

Some research models go further and listen while they talk. Moshi models its own speech and the user’s speech as parallel streams, so it can handle overlaps and interruptions without waiting for a clean end of turn.

Latency sets the design

Dhruv Batra treats live conversation as its own niche that needs its own kind of model. In math or coding, a model can take time to think. In a spoken exchange, he says, the budget is about a hundred milliseconds, and he expects voice agents to be built on specialized full-duplex audio models. Measure the delay from the end of the caller’s speech to the first audio back, not just the model’s response time. The general method is in how to reduce AI response latency.

Testing is harder than for text

Twilio’s Vinnie Giarrusso describes what changes for evaluation. A text request is one input and one output. A voice session means setting up a WebSocket connection and handling audio as it streams. You then have to check whether the stream came back correctly, whether an interruption was caught partway through, and what happened after it. He suggests starting with task-based evaluations, judging whether the call achieved its goal, before grading every turn of the stream.

How to choose

Illustrative example: a clinic wants an agent that books appointments by phone. A chained pipeline makes sense if every request must pass a written policy check before the agent answers, if the team needs a reviewed transcript of each turn, or if the clinic wants a specific synthetic voice. A speech-to-speech model makes sense if callers interrupt a lot and the main complaint is that the agent feels slow. Either way, the booking itself should be a tool call the agent makes, so the calendar stays the source of truth.

Where it falls short

Audio evaluation tooling is less mature than text evaluation, and a fluent voice can make a wrong answer sound confident. Speech-to-speech models give you fewer places to step in between hearing and speaking, so plan where your checks run before choosing one. For the broader set of models that see, hear and read, see what multimodal AI is.

Disclosure: Episode 59 is sponsored by Galileo, the Season 3 presenting sponsor; the thanks is at 1:22. Episodes 6 and 20 were sponsored by Galileo, where Conor then worked; in episode 6 he introduces himself as Galileo’s head of developer awareness at 0:00.