The voice agent stack: latency, turn-taking and barge-in

The pipeline

Every stage adds delay and every stage is separately replaceable, which is why the category has so many vendors — each one occupies a different subset of that chain. Knowing which stage a product covers is more useful than knowing its feature list.

Note that only one row is a decision rather than a computation. That is the row this article is mostly about, because it is both the largest contributor in most deployments and the one teams overlook — you cannot profile it, since nothing is running.

Where the latency actually is

The reason it exists is that humans pause mid-sentence. Someone saying “I need to reschedule my appointment… to next Tuesday” produces a silence in the middle that is not the end of their turn. Cut the wait too short and the agent answers the first half. Make it long enough to be safe and every exchange feels sluggish.

Frameworks expose it directly. LiveKit’s endpointing takes a min_delay and max_delay, and its Python SDK also offers dynamic endpointing — adapting the delay within that range based on the pause statistics of the current session. That is the right shape of solution: a caller who speaks in long deliberate phrases and one who fires short answers should not get the same fixed wait.

Turn detection modes

The choice is a latency-versus-naturalness dial, and unlike most such dials it has a clear default.

The model-based option is worth understanding rather than just enabling. A pure VAD decides on silence; a turn-detector model decides on whether the sentence sounds finished. “I’d like to book an appointment for” is acoustically complete and semantically obviously not, and only the second approach can tell.

Manual mode deserves a mention because it is under-used. For confirmations of consequential actions, an explicit turn boundary removes an entire class of error — and if the action needs approval anyway, the approval and the turn boundary can be the same interaction. That intersects with where the authority boundary sits.

Barge-in, and the backchannel problem

This is the single most common reason a voice agent feels broken to someone who has never used one, because backchannelling is involuntary. Callers do not know they are doing it, so from their side the agent simply keeps stopping for no reason.

Good frameworks treat it as a classification problem rather than a threshold. LiveKit’s interruption handling has an adaptive mode — the default on their cloud — described explicitly as distinguishing true interruptions from conversational backchannelling, alongside a plain vad mode that triggers on speech start/stop.

The last pair is the part worth copying even if you build your own. A misclassified backchannel that permanently silences the agent leaves the caller in dead air; the same misclassification with a resume path is a barely perceptible stumble. Recovering from being wrong is more valuable than being right slightly more often.

Two more controls worth setting

discard_audio_if_uninterruptible drops buffered audio while the agent is in a state that cannot be interrupted — without it, speech captured during an uninterruptible passage arrives afterwards and gets acted on late, which produces the agent answering something the caller said thirty seconds ago.

And a cap on caller turn length — a maximum word count or duration — with a handler for exceeding it. Without one, a caller who monologues produces an unbounded transcript and an agent that never gets a turn. The documented default behaviour on exceeding it is to generate a reply that cannot itself be interrupted, which is exactly right: the agent needs one uninterrupted sentence to regain the floor.

Repair

The failure mode to design against is confident action on a misheard value. An agent that books Tuesday when the caller said Thursday has done something worse than failing, because it will report success.

Note the interaction with turn detection: a read-back confirmation is exactly the moment a caller interrupts, because they heard the wrong value and reacted. So confirmation prompts should be interruptible even if other agent speech is not — the interruption there is a signal, not noise.

Telephony changes the numbers

LiveKit’s telephony-optimised example for a realtime model is concrete about the direction: a detection threshold of 0.7 described as less sensitive for noisy phone audio, prefix_padding_ms of 300, and silence_duration_ms of 400 described as a tighter window for snappier turn closing.

// Direction of travel for phone audio, not a universal setting:
//   raise the detection threshold  → fewer false triggers on line noise
//   tighten the silence window     → snappier turn closing
{
  threshold: 0.7,            // less sensitive than a clean-mic default
  prefix_padding_ms: 300,    // audio retained before detected speech
  silence_duration_ms: 400,  // silence before the turn is considered over
}

The caption matters. These are documented examples for one provider pairing, and the correct values depend on your carrier path, your codec, your callers and your domain. A collections call and a restaurant booking do not want the same patience.

One protocol-level constraint worth knowing before designing session handling: for conversational use, providers may require a single WebSocket per conversation, and voice settings can be fixed at connection time. Deepgram’s streaming TTS documents both, along with hard limits — 2,000 characters per request for Aura models, a throughput cap of 2,400 characters per minute, a flush limit of 20 in any 60 seconds, and a 60-minute connection timeout. Those are the numbers that break a long-running call, and none of them are latency.

Build on infrastructure or buy a platform

The honest observation is that most teams should start on a platform, and that the reason to leave is almost never a feature — it is that the thing making calls feel human turns out to be endpointing and interruption tuning, and at some point you need finer control of both than a platform exposes.

Which is worth knowing at the start, because it argues for keeping your agent logic, tools and governance outside whatever platform you pick. The conversational layer is the part you might replace; the tool, governance and evidence layers should not have to move with it.

Measure it yourself

Four measurements, all taken from the caller’s perspective rather than from inside a service:

The second is the one that redirects the work. Teams routinely spend a sprint on model latency and find the endpointing wait was three-quarters of the number — at which point a configuration change delivers more than the sprint did.

Percentiles rather than averages, because voice is judged on the bad calls. An agent that is fast on average and occasionally pauses for three seconds is experienced as an agent that pauses for three seconds, and the mean will tell you it is fine.

Finally, instrument the interruption counters as first-class metrics rather than debug logs. False interruptions and missed interruptions are the two numbers that most closely track whether callers find the thing pleasant, and neither appears in any latency dashboard.