AI Voice Agent Interruptions: How to Fix Barge-In Problems
A caller says 'yes' while an AI receptionist is talking. The agent stops mid-sentence. Seconds later, it starts again or repeats itself. In another call, the customer says 'No, Tuesday,' but the agent keeps talking and misses the correction. These are different failures and need different fixes.
Barge-in is when a person interrupts an agent's speech. A reliable system must recognize a meaningful interruption without treating every short acknowledgment or background sound as a request to stop.
First, identify the failure
| Symptom | Possible cause | Test |
|---|---|---|
| Agent stops on 'mm-hmm' | Backchannel classified as interruption | Speak short acknowledgments during a long response |
| Agent ignores a correction | Interruption threshold or turn logic too restrictive | Say 'Stop, change the date' mid-response |
| Agent repeats a sentence | Transcript/audio state was not aligned after interruption | Review played audio and conversation history |
| Agent resumes too late | False-interruption recovery delay | Compare timings after a cough or short noise |
| Agent hears itself | Echo or audio routing problem | Repeat test with headset and phone speaker |
These are diagnostic hypotheses, not proof of one particular root cause.
Step 1: Capture the actual call timeline
Record timestamps for user speech start, voice activity detection, end-of-turn decision, agent speech start, interruption decision, speech cancellation and resumed output. Capture redacted transcripts and audio only with appropriate notice, retention rules and privacy safeguards.
Without a timeline, changing model prompts is unlikely to isolate an audio/turn-taking bug.
Step 2: Separate backchannels from genuine corrections
A brief 'right' or 'okay' may be a listener cue; 'No, send it tomorrow' is an intentional correction. Test both against representative accents, languages, noisy environments and telephone codecs.
LiveKit's adaptive interruption handling documentation describes distinguishing intentional interruption from backchanneling through acoustic signals. It also notes that performance can vary by language. Do not assume one mode behaves the same across all callers.
Step 3: Tune the turn-taking controls
Inspect the actual SDK version and deployed configuration. Current LiveKit turn-taking guidance discusses interruption detection modes, minimum speech duration, false-interruption recovery and endpointing settings.
Make one change per test cycle. Shorter thresholds can increase responsiveness but may also increase false interruptions. More conservative thresholds may miss urgent corrections. Evaluate on recorded test scenarios rather than tuning only from a single call.
Step 4: Keep audio and conversation state aligned
When an interruption is accepted, stop the audio output promptly and ensure the model does not treat unplayed speech as if the customer heard it. The LiveKit turns overview documents history truncation to the portion heard before an interruption.
For tool actions, decide whether the interruption cancels only speech, a pending read, or an irreversible action. Canceling text-to-speech should not silently roll back a committed booking. If a transaction already happened, explain its state before proceeding.
Step 5: Run a repeatable test matrix
Run at least these scenarios as a suggested test suite, not an industry-mandated count:
- Customer says 'uh-huh' during a long explanation; agent should normally continue.
- Customer says 'Stop, I meant Friday'; agent should acknowledge and correct.
- Background typing and television are present.
- The agent is spelling a reference code while the user corrects one digit.
- Caller interrupts during a booking tool call.
- Caller switches language or speaks over the agent.
- Agent is briefly interrupted by cough or silence, then recovers naturally.
For each test, capture whether the user intent was recognized, whether speech stopped at the expected moment, whether the task state remained correct and whether the interaction recovered.
Step 6: Define measurable acceptance criteria
Track false interruptions per tested call, missed meaningful interruptions, correction completion rate, time to stop speaking, repeated speech incidents and booking-state errors. Set acceptable thresholds using your own target audience and service risk. No single latency figure is appropriate for every voice pipeline or telephony path.
An illustrative troubleshooting sequence
A receptionist keeps stopping when callers say 'okay.' First confirm echo is not feeding the agent its own audio. Then compare behavior with simple VAD and adaptive interruption detection in a controlled environment. Review whether false-interruption recovery restores speech cleanly. Recheck legitimate corrections before accepting any reduction in false triggers.
This is an illustrative sequence, not evidence of an actual customer deployment.
Next step
Build a call replay suite from realistic, permissioned examples, and rerun it after audio, VAD, model, SDK or telephony changes. Pair interruption testing with LuMay's voice agent load testing checklist.
Frequently asked questions
Should every caller sound interrupt the agent?
No. Short acknowledgments and background noises are not necessarily requests to change the conversational turn.
Can a better prompt solve barge-in failures?
Prompts can improve wording and recovery, but turn detection and audio playback usually require pipeline-level investigation.
Does stopping speech cancel a tool action?
Not automatically. Speech interruption and business action cancellation need an explicit state-management design.
