Most voice agents decide you are done talking by measuring silence. Voice activity detection flags the audio as speech or not speech, a timer runs during the not-speech, and when the timer crosses a threshold the turn ends and generation begins.
It is a reasonable first architecture. It is also wrong in a way that gets worse the more natural the conversation gets.
The failure looks like this. Someone is thinking while they speak. They pause mid-sentence, because that is what people do when they are constructing a thought rather than reading one out. The pause crosses the threshold. The system ends the turn and starts responding, and the user gets cut off by a machine, mid-thought.
The fix that presents itself immediately is to raise the threshold. Wait longer before deciding the turn is over.
This is the part worth writing down: raising the threshold trades one failure for a worse one. Latency in conversation is not a smooth cost curve. Below a certain point people do not perceive a delay at all. Above it, they perceive a system that is slow, and slowness in a voice interface reads as broken in a way that visual latency does not, because there is no spinner, no loading state, nothing to indicate the machine is alive. Silence from a voice agent is indistinguishable from failure. So you push the threshold up to stop the interruptions, and you lose the thing that made the product feel real.
Both ends of the tuning range are bad. That is the actual shape of the problem, and it is why threshold tuning is not the answer at any value.
The reframe that helped: silence is the weakest available signal for a decision this important, and it is the only one a pure VAD approach is using. Humans do not detect turn ends by timing gaps. We use syntax, whether the clause has resolved. Prosody, whether the pitch fell into a terminal contour or stayed suspended. Semantics, whether the thing said is complete enough to answer. We are running a continuous prediction of whether the other person is finished, and we start preparing our response before they actually stop.
Which points at a different architecture. Instead of a timer that fires on silence, run a lightweight endpointing model over the partial transcript that outputs a probability the turn is complete, and let silence be one input to that rather than the whole decision. A trailing-off pause after a resolved clause and a pause mid-clause are the same event to a VAD and completely different events to anything that reads the text.
The second piece is decoupling detection from commitment. A turn-end prediction does not have to be a single irreversible decision. You can begin generating on a probable endpoint and cancel cheaply if the user resumes. That converts a hard classification problem into a soft one, and soft problems tolerate being wrong.
None of this is solved. Barge-in handling, cancellation cost, and what happens when the user talks over the response are all still open on my end, and I would rather write about them while they are open than after I have tidied them into a conclusion.
If you have shipped real-time voice, I want to hear what your endpointing actually looks like in production, particularly what you do when the prediction fires early and generation has already started.
Top comments (1)
"Both ends of the tuning range are bad" is the correct framing and it is the thing that took me longest to accept, because raising the threshold does make the immediate complaint go away.
The piece I would add sits under your second point, decoupling detection from commitment. Cancellation being cheap is not a property of your architecture, it is a property of your audio path, and it is usually much more expensive than it looks. When we cancelled, generation stopped almost immediately and the caller kept hearing the agent for a median of 1.85 seconds, because the TTS frame was already committed, the packets were already sent, and the client's playout buffer drained to completion before it went quiet. Server-side stopped and caller-side stopped were nearly two seconds apart.
That matters for your design specifically: if you generate on a probable endpoint and plan to cancel when you are wrong, the cost of being wrong is the tail, not the tokens. We got ours down by flushing the client buffer explicitly instead of letting it drain, and by shrinking the TTS frame from 400 ms to 120. Until that landed, soft endpointing would have made things worse rather than better, because we were generating more often and unable to take it back.
Worth measuring before you build the model: the gap between your VAD firing and the caller's device actually going silent.