Voice agents represent the frontier of AI interaction - humans speaking naturally with AI systems. The challenge isn't just speech recognition and synthesis, it's achieving natural conversation flow with sub-800ms latency while handling interruptions, background noise, and emotional nuance. This skill covers two architectures: speech-to-speech (OpenAI Realtime API, lowest latency, most natural) and pipeline (STT→LLM→TTS, more control, easier to debug). Key insight: latency is the constraint. Hu
Voice agents represent the frontier of AI interaction - humans speaking
naturally with AI systems. The challenge isn't just speech recognition
and synthesis, it's achieving natural conversation flow with sub-800ms
latency while handling interruptions, background noise, and emotional
nuance.
This skill covers two architectures: speech-to-speech (OpenAI Realtime API,
lowest latency, most natural) and pipeline (STT→LLM→TTS, more control,
easier to debug). Key insight: latency is the constraint. Humans expect
responses in 500ms. Every millisecond matters.
84% of organizations are increasing voice AI budgets in 2025. This is the
year voice agents go mainstream.
Principles
Latency is the constraint - target <800ms end-to-end
Jitter (variance) matters as much as absolute latency
VAD quality determines conversation flow
Interruption handling makes or breaks the experience
Start with focused MVP, iterate based on real conversations
Vapi - When: Managed voice agent platform Note: No infrastructure management
Retell AI - When: Low-latency voice agents Note: Best context preservation on interruption
Patterns
Speech-to-Speech Architecture
Direct audio-to-audio processing for lowest latency
When to use: Maximum naturalness, emotional preservation, real-time conversation
SPEECH-TO-SPEECH ARCHITECTURE:
"""
[User Audio] → [S2S Model] → [Agent Audio]
Advantages:
Lowest latency (sub-500ms)
Preserves emotion, emphasis, accents
Most natural conversation flow
Disadvantages:
Less control over responses
Harder to debug/audit
Can't easily modify what's said
"""
OpenAI Realtime API
"""
import { RealtimeClient } from '@openai/realtime-api-beta';
const client = new RealtimeClient({
apiKey: process.env.OPENAI_API_KEY,
});
// Configure for voice conversation
client.updateSession({
modalities: ['text', 'audio'],
voice: 'alloy',
input_audio_format: 'pcm16',
output_audio_format: 'pcm16',
instructions: You are a helpful customer service agent. Be concise and friendly. If you don't know something, say so rather than making things up.,
turn_detection: {
type: 'server_vad', // or 'semantic_vad'
threshold: 0.5,
prefix_padding_ms: 300,
silence_duration_ms: 500,
},
});
// Pipe audio to transcription
audioStream.pipe(transcription);
}
"""
Optimization Tips:
Start TTS while LLM still generating (streaming)
Pre-compute first response segment during user speech
Use Flash/turbo models for latency
Voice Activity Detection Pattern
Detect when user starts/stops speaking
When to use: All voice agents need VAD for turn-taking
VOICE ACTIVITY DETECTION (VAD):
"""
VAD Types:
Energy-based: Simple, fast, noise-sensitive
Model-based: Silero VAD, more accurate
Semantic VAD: Understands meaning, best for conversation
"""
Silero VAD (Popular Open Source)
"""
import { SileroVAD } from '@pipecat-ai/silero-vad';
const vad = new SileroVAD({
threshold: 0.5, // Speech probability threshold
min_speech_duration: 250, // ms before speech confirmed
min_silence_duration: 500, // ms of silence = end of turn
});
vad.on('speech_start', () => {
console.log('User started speaking');
// Stop any playing TTS (barge-in)
audioPlayer.stop();
});
// Feed audio to VAD
audioStream.on('data', (chunk) => {
vad.process(chunk);
});
"""
OpenAI Semantic VAD
"""
// In Realtime API session config
client.updateSession({
turn_detection: {
type: 'semantic_vad', // Uses meaning, not just silence
// Model waits longer after "ummm..."
// Responds faster after "Yes, that's correct."
},
});
"""
Barge-In Handling
"""
// When user interrupts:
function handleBargeIn() {
// 1. Stop TTS immediately
audioPlayer.stop();
// VAD triggers barge-in
vad.on('speech_start', () => {
if (audioPlayer.isPlaying) {
handleBargeIn();
}
});
"""
Latency Optimization Pattern
Achieving <800ms end-to-end response time
When to use: Production voice agents
LATENCY OPTIMIZATION:
"""
Target Metrics:
End-to-end: <800ms (ideal: <500ms)
Time-to-First-Token (TTFT): <300ms
Barge-in response: <200ms
Jitter variance: <100ms std dev
"""
Pipeline Latency Breakdown
"""
Typical breakdown:
VAD processing: 50-100ms
STT first result: 150-200ms
LLM TTFT: 100-300ms
TTS TTFA: 75-200ms
Audio buffering: 50-100ms
Total: 425-900ms
"""
Optimization Strategies
1. Streaming Everything
"""
// Stream STT results as they come
stt.on('partial_transcript', (text) => {
// Start processing before final transcript
llmPreprocessor.prepare(text);
});
"""
// Run inference closer to user
// - Cloud regions near user
// - Edge computing for VAD/STT
// - WebSocket over HTTP for lower overhead
"""
Conversation Design Pattern
Designing natural voice conversations
When to use: Building voice UX
CONVERSATION DESIGN:
Voice-First Principles
"""
Voice is different from text:
No undo button - say it right the first time
Linear - user can't scroll back
Ephemeral - easy to miss information
Emotional - tone matters as much as words
"""
Response Design
"""
Keep responses short (10-20 seconds max)
Front-load the answer
Use signposting for lists
Bad: "I found several options. The first is... second is..."
Good: "I found 3 options. Want me to go through them?"
Confirm understanding
Bad: "I'll transfer $500 to John."
Good: "So that's $500 to John Smith. Should I proceed?"
"""
Prompting for Voice
"""
system_prompt = '''
You are a voice assistant. Follow these rules:
Be concise - keep responses under 30 words
Use natural speech - contractions, casual language
Never use formatting (bullets, numbers in lists)
Spell out numbers and abbreviations
End with a question to keep conversation flowing
If unclear, ask for clarification
Never say "I'm an AI" unless asked
Good: "Got it. I'll set that reminder for three pm. Anything else?"
Bad: "I have set a reminder for 3:00 PM. Is there anything else I can assist you with today?"
'''
"""
Error Recovery
"""
// Handle recognition errors gracefully
const errorResponses = {
no_speech: "I didn't catch that. Could you say it again?",
unclear: "Sorry, I'm not sure I understood. You said [repeat]. Is that right?",
timeout: "Still there? I'm here when you're ready.",
};
// Always offer human fallback for complex issues
if (confidenceScore < 0.6) {
response = "I want to make sure I get this right. Would you like to speak with a human agent?";
}
"""
Sharp Edges
Response Latency Exceeds 800ms
Severity: CRITICAL
Situation: Building a voice agent pipeline
Symptoms:
Conversations feel awkward. Users repeat themselves. "Are you
there?" questions. Users hang up or give up. Low satisfaction
scores despite correct answers.
Why this breaks:
In human conversation, responses typically arrive within 500ms.
Anything over 800ms feels like the agent is slow or confused.
Users lose confidence and patience. Every component adds latency:
VAD (100ms) + STT (200ms) + LLM (300ms) + TTS (200ms) = 800ms.
Recommended fix:
Measure and budget latency for each component:
Target latencies:
VAD processing: <100ms
STT time-to-first-token: <200ms
LLM time-to-first-token: <300ms
TTS time-to-first-audio: <150ms
Total end-to-end: <800ms
Optimization strategies:
Use low-latency models:
STT: Deepgram Nova-3 (150ms) vs Whisper (500ms+)
TTS: ElevenLabs Flash (75ms) vs standard (200ms+)
LLM: gpt-4o-mini streaming
Stream everything:
Don't wait for full STT transcript
Stream LLM output to TTS
Start audio playback before TTS finishes
Pre-compute:
While user speaks, prepare context
Generate opening phrase in parallel
Edge deployment:
Run VAD/STT at edge
Use nearest cloud region
Measure continuously:
Log timestamps at each stage, track P50/P95 latency
Response Time Variance Disrupts Rhythm
Severity: HIGH
Situation: Voice agent with inconsistent response times
Symptoms:
Conversations feel unpredictable. User doesn't know when to speak.
Sometimes agent responds immediately, sometimes after long pause.
Users talk over agent. Agent talks over users.
Why this breaks:
Jitter (variance in response time) disrupts conversational rhythm
more than absolute latency. Consistent 800ms feels better than
alternating 400ms and 1200ms. Users can't adapt to unpredictable
timing.
Symptoms:
Agent interrupts user mid-thought. Or waits too long after user
finishes. "Let me think..." triggers premature response. Short
answers have awkward pause before response.
Why this breaks:
Simple silence detection (e.g., "end turn after 500ms silence")
doesn't understand conversation. Humans pause mid-sentence.
"Yes." needs fast response, "Well, let me think about that..."
needs patience. Fixed timeout fits neither.
Recommended fix:
Use semantic VAD:
OpenAI Semantic VAD:
client.updateSession({
turn_detection: {
type: 'semantic_vad',
// Waits longer after "umm..."
// Responds faster after "Yes, that's correct."
},
});
Pipecat SmartTurn:
const pipeline = new Pipeline({
vad: new SileroVAD(),
turnDetection: new SmartTurn(),
});
if (endsWithComplete && !hasFillers) {
return 300; // Fast response
} else if (hasFillers) {
return 1500; // Wait for continuation
}
return 700; // Default
}
Agent Doesn't Stop When User Interrupts
Severity: HIGH
Situation: User tries to interrupt agent mid-sentence
Symptoms:
Agent talks over user. User has to wait for agent to finish.
Frustrating experience. Users give up and abandon call.
"STOP! STOP!" doesn't work.
Why this breaks:
Without barge-in handling, the TTS plays to completion regardless
of user input. This violates basic conversational norms - in human
conversation, we stop when interrupted.
if (isClarification(firstWords)) {
// "What?", "Sorry?" - repeat last sentence
repeatLastSentence();
} else {
// Real interruption - stop and listen
handleFullInterruption();
}
});
Response time target:
Barge-in response: <200ms
User should feel heard immediately
Generating Text-Length Responses for Voice
Severity: MEDIUM
Situation: Prompting LLM for voice agent responses
Symptoms:
Agent rambles. Users lose track of information. "Can you repeat
that?" requests. Users interrupt to ask for shorter version.
Low comprehension of conveyed information.
Why this breaks:
Text can be scanned and re-read. Voice is linear and ephemeral.
A 3-paragraph response that works in chat is overwhelming in voice.
Users can only hold ~7 items in working memory.
Recommended fix:
Constrain response length in prompts:
system_prompt = '''
You are a voice assistant. Keep responses UNDER 30 WORDS.
For complex information, break into chunks and confirm
understanding between each.
Instead of: "Here are the three options. First, you could...
Second... Third..."
Say: "I found 3 options. Want me to go through them?"
Never list more than 3 items without pausing for confirmation.
'''
if (information.length > 3) {
response = I have ${information.length} items. Let's go through them one at a time. First: ${information[0]}. Ready for the next?;
}
Progressive disclosure:
"I found your account. Want the balance, recent transactions, or something else?"
// Don't dump all info at once
Using Bullets/Numbers/Markdown in Voice
Severity: MEDIUM
Situation: Formatting LLM output for voice
Symptoms:
"First bullet point: item one" read aloud. Numbers read as "one
two three" instead of "one, two, three." Markdown artifacts in
speech. Robotic, unnatural delivery.
Why this breaks:
TTS models read what they're given. Text formatting intended for
visual display sounds robotic when read aloud. Users can't "see"
structure in audio.
Recommended fix:
Prompt for spoken format:
system_prompt = '''
Format responses for SPOKEN delivery:
No bullet points, numbered lists, or markdown
Spell out numbers: "twenty-three" not "23"
Spell out abbreviations: "United States" not "US"
Use verbal signposting: "There are three things. First..."
Never use asterisks, dashes, or special characters
'''
Symptoms:
"I didn't catch that" frequently. Background noise triggers
false starts. Fan/AC causes continuous listening. Car engine
noise confuses STT.
Why this breaks:
Default VAD thresholds work for quiet environments. Real-world
usage includes background noise that triggers false positives
or masks speech, causing false negatives.
// Prevent agent's voice from being transcribed
const echoCanceller = new EchoCanceller();
echoCanceller.reference(ttsOutput);
const cleanedAudio = echoCanceller.process(userAudio);
STT Produces Incorrect or Hallucinated Text
Severity: MEDIUM
Situation: Processing unclear or accented speech
Symptoms:
Agent responds to something user didn't say. Names consistently
wrong. Technical terms misheard. "I said X, not Y" frustration.
Why this breaks:
STT models can hallucinate, especially on proper nouns, technical
terms, or accented speech. These errors propagate through the
pipeline and produce nonsensical responses.
