Overview
The Voice Engine is the real-time speech pipeline on top of the Core Engine: it receives audio via Twilio, transcribes it with Deepgram, passes the transcript to the reasoning runtime, synthesises a response with ElevenLabs or Deepgram TTS, and streams the audio back. For evaluation, the useful frame is the hand-off boundary at each stage - where speech recognition ends and reasoning begins, and where reasoning ends and synthesis begins - because each boundary is where a misrecognition or a slow tool call can show up as a bad caller experience.
Call evidence captured below every stream
Call Flow
- Call arrives - Twilio opens a WebSocket to the Voice Engine.
- Transcription - audio streams through Deepgram STT, returning partial and final transcriptions.
- Reasoning - the transcript is passed to the Core Engine with conversation history; the LLM reasons about intent and may emit tool calls.
- Mid-call tool calling - tool calls route to connectors while the call is live (e.g. a record lookup).
- Synthesis - the LLM's reply is synthesised to audio via ElevenLabs or Deepgram TTS.
- Call ends - transcript, outcome tags, and sentiment are written to the database; a full trace is persisted for the session.
Integration Dependencies
- Telephony - Twilio handles inbound/outbound call delivery; your existing phone numbers need to be configured to route through it.
- STT/TTS providers - Deepgram and ElevenLabs are the configured providers; confirm language and accent coverage directly against your caller population rather than assuming universal coverage.
- Connector latency - mid-call tool calls depend on the connected system's response time; a slow external API shows up as caller-perceived latency.
- Human handoff target - a receiving human agent/queue must be configured for transfers to work, with transcript and context passed along.
Evaluation Scenarios Worth Testing
- Interruptions - does the pipeline handle a caller talking over the agent's response cleanly?
- Silence - what happens if the caller goes silent mid-call - does the agent prompt, wait, or disconnect?
- Misrecognised speech - how does the flow behave when STT returns a low-confidence or incorrect transcript?
- Handoff - does the receiving human agent actually get the transcript and context, or start cold?
Illustrative Example
The following is illustrative, not a specific deployment.
An inbound caller asks about an order status mid-sentence while background noise is present. The transcript comes back with a confidence flag; the flow is configured to ask a clarifying question rather than guess at the order number. Once confirmed, the agent looks up the record via a connector call live on the call, and reads the status back before the caller would have expected a human to finish typing the same lookup.
FAQ
What languages does the voice pipeline support?
This page does not assert specific language or accent coverage figures - confirm directly against your caller population and the configured STT/TTS providers.
What is typical turn-taking latency?
The root platform page describes the pipeline as optimised for sub-second turns on simple exchanges; actual latency depends on the complexity of the reasoning step and any connector calls in the loop - confirm with a live test for your use case rather than assuming a fixed number.
Can the agent transfer to a human mid-call?
Yes - human handoff passes the full transcript and a context summary so the receiving agent isn't starting cold.
Does recording happen automatically?
Call recordings are stored and linked to the call session record in the database when recording is enabled for the agent.
How is voice agent performance measured?
See Analytics & ROI for containment, sentiment, and connector-success metrics sourced from voice sessions.
Related
- Core Engine - the reasoning runtime the Voice Engine sits on
- Connectors & Adapters - the systems agents call mid-call
- Analytics & ROI - KPIs measuring voice agent performance
- UK AI Platform hub - all 8 component pages