AI Voice Processing Pipeline Documentation

Understand AI voice workflows including speech recognition, LLM orchestration, text-to-speech, and analytics.

Pipeline Architecture

Every Voxentis voice agent runs on a real-time processing pipeline. Audio flows through a sequence of components: Speech-to-Text â†’ Language Model â†’ Text-to-Speech. You can customize each component and add middleware between stages.

Pipeline stages execute sequentially for each conversational turn
Middleware can intercept and transform data between stages
Each stage can have provider-specific settings
Pipelines support hot-swapping components without downtime

Full Pipeline Config

{
  "pipeline": {
    "stages": [
      {
        "type": "stt",
        "provider": "deepgram",
        "config": {
          "model": "nova-2",
          "language": "en-US",
          "interim_results": true,
          "vad_events": true
        }
      },
      {
        "type": "middleware",
        "name": "intent_classifier",
        "config": {
          "model": "voxentis/intent-v2",
          "confidence_threshold": 0.8
        }
      },
      {
        "type": "llm",
        "provider": "openai",
        "config": {
          "model": "gpt-4o",
          "temperature": 0.7,
          "max_tokens": 1024,
          "system_prompt": "You are a helpful voice assistant..."
        }
      },
      {
        "type": "middleware",
        "name": "response_filter",
        "config": {
          "max_length": 200,
          "remove_markdown": true
        }
      },
      {
        "type": "tts",
        "provider": "elevenlabs",
        "config": {
          "voice_id": "pNInz6obpgDQGcFmaJgB",
          "stability": 0.5,
          "optimize_streaming_latency": 3
        }
      }
    ],
    "turn_taking": {
      "mode": "server_vad",
      "silence_threshold_ms": 700,
      "interruption_enabled": true
    }
  }
}

Stage Types

Stage	Description
stt	Converts caller audio to text. Runs continuously during the call, producing interim and final transcripts.
llm	Processes the transcript and generates a response. Receives the full conversation history and any tool results.
tts	Converts the LLM response to audio. Streams audio frames as the LLM generates tokens for minimal latency.
middleware	Custom processing between stages. Use for intent classification, response filtering, sentiment analysis, PII redaction, or logging.

Update Pipeline

curl -X PUT https://api.voxentis.com/v1/agents/agt_9f2k4h1a/pipeline \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "pipeline": {
      "stages": [...],
      "turn_taking": { "mode": "server_vad", "silence_threshold_ms": 500 }
    }
  }'

Pipeline changes take effect for new calls immediately. Active calls continue with their original pipeline configuration.