October 14, 2026 · LuMay AI
Test an AI voice agent by placing repeatable real telephone calls that inject interruptions, delays, and transfer faults, then recording percentile latency, call state, and human-acceptance evidence against pass/fail criteria you set for your workload.
Key takeaways
- Measure from the caller's side. Latency starts when the caller stops speaking, not when the model receives text.
- Test cancellation, not just stopping. Audio, queued speech, pending tools, and conversation state must all reset.
- Report percentiles with sample counts. Give p50, p95, and p99 per scenario, run cold, warm, and under concurrency.
- A human proves a transfer, not an API. Require bridged audio, acceptance, and authorized context.
- Thresholds are workload-specific. Set them from your own baseline and risk.
What you are testing
- Barge-in: a caller speaking while the agent talks. A pass means playback stops, the new speech becomes the next turn, and state stays correct.
- Turn detection (endpointing): deciding the caller has finished. Too eager cuts people off; too slow creates dead air.
- End-to-end latency: time from the caller's last word to the first audible agent sound. Model "time to first token" is only one slice.
- Warm transfer: the agent briefs a human before connecting the caller. LiveKit's version holds the caller, briefs the human in a private room, then connects both.
- Blind (cold) transfer: the call is forwarded and the agent drops out, often through SIP REFER (a Session Initiation Protocol forwarding request).
Provider differences to record
Behavior varies by provider and version. These points were checked against vendor documentation on October 14, 2026; recheck before each release.
- Twilio ConversationRelay: the TwiML reference lists interruption sensitivity (default high), a backchannel filter (default off), and a tokens-played event. Input reported during agent speech changed from any to none in May 2025.
- Twilio speech-to-text (STT): Deepgram is the default, but accounts that used the product before September 12, 2025 default to Google. The speechTimeout setting behaves differently with Deepgram's Flux model.
- LiveKit Agents: adaptive interruption handling needs LiveKit Cloud or dev mode, Python SDK 1.5.0 or Node.js SDK 1.2.0, and a compatible setup. Otherwise it falls back to voice activity detection (VAD) alone.
- LiveKit realtime models: OpenAI Realtime can work with server-side turn detection off; Gemini Live cannot use the adaptive gate.
- LiveKit warm transfer: the prebuilt task is Python-only and in beta.
Set up the test harness
- Synthetic data only: invented names, numbers, and orders. Never real caller records.
- Approved recording: agree consent, retention, and access in writing before recording any call.
- Representative audio: accents, devices (handset, speakerphone, Bluetooth), and noise (street, kitchen, open office).
- Real telephone calls: web or text tests miss codec, jitter, and echo behavior.
- Repeated runs: one pass proves nothing; repeat each scenario enough to show spread.
- Versioned configurations: log prompt, model, voice, speech providers, endpointing, and flow versions per run.
- Synchronized evidence: caller-side audio, agent logs, tool logs, and carrier records on one clock.
The test matrix (16 tests)
Run each test across your audio conditions. Numeric limits are workload-specific, so set your own.
| # | Caller action or fault | Expected behavior | Pass/fail criterion | Evidence |
|---|---|---|---|---|
| I1 | Says "Stop" mid-sentence | Speech halts; agent asks what is needed | Audio stops within your limit; sentence never resumes | Caller-side audio, playback log |
| I2 | Corrects a detail mid-sentence | Agent uses the correction | Stored value equals the correction | Transcript, tool payload |
| I3 | Talks over agent for several seconds | Agent yields, then hears the whole turn | No lost words; no overlap after yield | Dual-channel recording |
| I4 | Says "mm-hm" while agent talks | Agent continues, if set to ignore | No cancelled reply | Audio, interruption events |
| I5 | Background noise only (TV, door slam) | No false barge-in | False cancellations within your tolerance | Labelled noise recordings |
| I6 | Silence after a question | Bounded re-prompts, then polite exit or handoff | Re-prompts at or below policy | Call log, audio |
| I7 | Interrupts three times while a booking tool runs | Pending action completes once or cancels cleanly | One booking; caller told the outcome | Tool log, booking record |
| L1 | Short answer after idle period (cold) | First audible response | p50, p95, p99 within your target | Caller-side timestamps |
| L2 | Same prompt repeated back to back (warm) | Stable response | Warm percentiles within target; cold gap reported | Timestamps, build ID |
| L3 | Expected peak concurrent calls | Latency and completions hold | Shift versus baseline within tolerance; no failed calls | Load log, carrier records |
| H1 | Human requested; staff busy or no answer | Caller informed; callback offered | Bounded retries; named owner holds task | Dial status, task record |
| H2 | Transfer number invalid or fails | Graceful fallback | No dead air; fallback owner assigned | Carrier record, audio |
| H3 | Voicemail answers the transfer | Not treated as a person | Caller not bridged to voicemail; fallback triggers | Detection result, audio |
| H4 | Human answers; context missing or excessive | Brief summary, authorized fields only | Human confirms summary; no unauthorized data | Summary log, confirmation |
| H5 | Caller hangs up during transfer | Clean teardown | No orphan task; callback only if appropriate | Call states, task log |
| H6 | Callback delayed or triggered twice | One callback, on time | No duplicate; delay within policy | Scheduler log, call records |
Interruption tests
Cover "Stop," mid-sentence corrections, overlapping speech, backchannels (short cues such as "mm-hm"), noise, silence, and repeated interruptions. For each, check five things.
- Playback cancellation: confirm audio stops at the caller's ear, not just in logs. Buffered audio can keep playing.
- Conversation state: the agent should know what was actually spoken before the cut-off. Check whether your platform reports it.
- Pending tools: an interrupted turn may leave a tool call running. Verify it completes once, cancels, or reconciles.
- Duplicate actions: repeat interruptions around every write action, then count records created.
- False versus missed barge-ins: track both. Tuning one often worsens the other.
Latency tests
Measure from the caller's side: the last caller speech to the first audible agent audio in a recording. Model time to first token excludes endpointing, tools, synthesis, and network, so it understates what callers experience.
- Percentiles and counts: report p50, p95, and p99 (values that 50%, 95%, and 99% of calls met or beat) with sample counts. A p99 from 50 calls is one or two slow calls.
- Cold and warm: separate first calls after idle from repeated calls.
- Concurrency: repeat at expected peak, then above it.
- Targets: set them from your baseline and caller tolerance. LuMay's latency page says it publishes no fixed benchmarks because results depend on configuration.
| Stage | Capture |
|---|---|
| Endpointing | Silence wait before the turn closes |
| Speech recognition | Time to final transcript |
| Model | Time to first token, plus any tool loop |
| Tools | Duration of each external call |
| Speech synthesis | Time to first audio chunk |
| Network and telephony | Caller-side arrival minus agent send time |
Do not add stage percentiles together. Sum the stages for each call, then take percentiles of those totals.
Handoff tests
An API acknowledgment or an answered call alone does not prove a successful human handoff. Require all six checks.
- Bridged audio: caller and human hear each other, confirmed on both legs.
- Human acceptance: a person confirms they took the call, not a voicemail or auto-attendant.
- Authorized context: the summary carries only fields the human may see. LuMay's security page describes PII masking and human approval gates as examples of this control.
- Bounded retries: cap attempts and ring time. Twilio's Dial timeout defaults to 30 seconds, and Twilio adds a five-second buffer.
- Caller updates: tell the caller what is happening during hold, retry, and failure.
- Fallback ownership: if nobody accepts, a named person or queue owns the callback.
Dial reports completed when the called party answered and connected, with busy, no-answer, failed, and canceled for other outcomes.
Voicemail can still answer. Answering machine detection can return human, machine, or unknown, and the docs warn it is not always right.
In LiveKit's warm flow, an unavailable human lets the agent return to the caller, explain, then retry or end.
Troubleshooting
| Symptom | Likely cause | First check |
|---|---|---|
| Agent talks over caller | Endpointing too slow, or interruption off | Interruption and VAD settings |
| Caller cut off early | Endpointing too aggressive | Silence delay |
| Stops for "mm-hm" | Backchannel not filtered | Backchannel or adaptive mode |
| Dead air on tool turns | Slow tool, no filler | Tool timings |
| Fast in lab, slow on calls | Network, codec, region | Carrier-side timestamps |
| Transfer "succeeds," caller hears silence | No bridge, or voicemail | Audio on both legs, dial status |
Test-record template
- Identity: run ID, date and time, tester, scenario ID.
- Configuration: prompt, model, voice, speech providers, endpointing, flow versions.
- Conditions: audio condition, device, call path, cold or warm, concurrency.
- Result: pass or fail, timestamps, evidence links.
- Triage: severity, owner, linked fix, retest date.
Release gates by severity
These gates are suggestions; calibrate them to your workload and risk.
| Severity | Example failure | Suggested rule |
|---|---|---|
| Sev 1 | Unauthorized data shared; emergency not escalated; irreversible duplicate action | Block release; none open |
| Sev 2 | Caller stranded after failed handoff; duplicate reversible action | Block until fixed and retested |
| Sev 3 | p95 above your target; false barge-ins above tolerance | Ship only with owner, deadline, and alert |
| Sev 4 | Awkward wording; minor audio flaws | Backlog |
Regression testing and monitoring
- Rerun triggers: any change to prompt, model, voice, speech providers, endpointing, telephony, or tools reruns the full matrix.
- Golden audio set: keep fixed recordings so versions compare fairly.
- Every failure becomes a test: add production incidents with their evidence.
- Production monitoring: track p95 and p99 by hour, transfer outcomes, duplicate actions, and sampled false barge-ins.
- Simulation plus people: automated callers give repeatable coverage; reviewers judge tone, clarity, and odd cases. Review real calls weekly.
Illustrative scenarios
- Restaurant: a caller changes party size while the booking tool runs. Pass: one reservation exists with the final size.
- Healthcare: a caller describes urgent symptoms. Pass: the agent follows your approved escalation script, transfers promptly, and shares only authorized context.
- Legal: a prospective client asks for advice. Pass: the agent declines advice, captures intake, and hands a summary to an attorney.
How LuMay Voice Agents fit this plan
LuMay Voice Agents (Voxentis) list capabilities that map to this plan. Verify each in your own test environment.
- Playback and records: the product page lists call playback; the features page lists transcripts, summaries, and analytics.
- Escalation: transfers carry a live summary, with configurable triggers such as sentiment, unrecognized intent, VIP flags, and keywords.
- Connectors: the integrations page lists Twilio, Google Calendar, Salesforce, Zendesk, plus REST and webhook options.
- Guardrails: custom guardrails and controlled PII handling are listed.
Bring this matrix to a demo and ask which timestamps, recordings, and logs you can export. Book a LuMay Voice Agent demo.
