Latency and Real-Time Optimization

Updated November 2, 2026 — Audio models. Several models in this course are deprecated: tts-1, tts-1-hd, and gpt-4o-mini-tts will stop on January 6, 2027 (successor: gpt-realtime-2.1-mini, which the OpenAI TTS guide states is not supported by /v1/audio/speech: voice synthesis migration moves to the Realtime API); gpt-realtime, gpt-4o-realtime, and legacy audio models on January 20, 2027 (gpt-realtime-2.1, gpt-realtime-2.1-mini, gpt-audio-1.5); whisper-1, gpt-4o-transcribe, and /v1/audio/translations on February 26, 2027 (gpt-transcribe, gpt-live-transcribe, v1/realtime/translations). For full-duplex voice, GPT-Live 1 has been available in the API since September 10, 2026.

Corrected November 3, 2026. The durations assigned to each step (“VAD 300-700 ms”, “network 20-100 ms”, “model 200-800 ms”) and perception thresholds (“beyond 500 ms”) lacked sources: OpenAI does not publish such figures, and the lesson now teaches you how to measure them. The instruction to place your server “in the same region as OpenAI servers (US East)” was removed due to lack of sourcing. Documented additions: semantic_vad, WARP to accelerate WebRTC session establishment. Sources: Voice activity detection · WebRTC with WARP · Cost optimization.

Latency and Real-Time Optimization

Latency is the number one enemy of a voice agent: the silence following the end of a question is immediately noticeable on the phone or in an application. This lesson details the sources of latency, how to measure them, and techniques to reduce them.

Anatomy of Latency

The total latency of a voice agent is the sum of several components:

Depending on your architecture, add the tools the agent calls and the session setup time. None of these durations are fixed: they depend on your settings, conversation length, and network path. The only way to know where the time goes is to time each step, as the monitor presented below does.

Optimizing VAD Detection

With server_vad, the response cannot start before the silence defined by silence_duration_ms ends: this is therefore a direct setting for perceived latency, to be balanced against the risk of interrupting the user. The values below are starting points to test, not OpenAI recommendations:

# Aggressive configuration (fast response, risk of cutting off user)
fast_vad = {
    "type": "server_vad",
    "threshold": 0.6,           # Higher threshold = fewer false positives
    "prefix_padding_ms": 200,
    "silence_duration_ms": 400  # Fast reaction
}

# Patient configuration (fewer interruptions, more latency)
patient_vad = {
    "type": "server_vad",
    "threshold": 0.4,
    "prefix_padding_ms": 400,
    "silence_duration_ms": 800
}

# Alternative: semantic VAD, which judges sentence end based on words;
# eagerness "high" responds ASAP, "low" lets the user finish
semantic_vad = {"type": "semantic_vad", "eagerness": "high"}

# Adaptive configuration based on context
def get_vad_config(context: str) -> dict:
    configs = {
        "menu_ivr": {"threshold": 0.6, "silence_duration_ms": 400},
        "conversation": {"threshold": 0.5, "silence_duration_ms": 600},
        "dictation": {"threshold": 0.3, "silence_duration_ms": 1200},
    }
    config = configs.get(context, configs["conversation"])
    return {
        "type": "server_vad",
        "prefix_padding_ms": 300,
        **config
    }

Optimizing System Instructions

Concise instructions reduce the context to process. Also remember that the Realtime API applies prompt caching automatically: stable instructions and tools, placed at the top and never modified during a session, are reused from turn to turn.

# Verbose instructions (more tokens to process)
bad_instructions = """
You are an intelligent and competent voice assistant working
for Corsen AI, a French company specializing in
artificial intelligence. You must answer user questions
professionally, concisely, and in French.
You have access to various tools to consult appointments,
check availability, and make reservations.
Do not hesitate to ask for clarification if the request is unclear.
"""

# Optimized instructions (same effect, fewer tokens)
good_instructions = """
Corsen AI voice assistant. French, concise, professional.
Use tools for appointments and reservations.
Ask for clarification if needed.
"""

Reducing Network Latency

Measuring Round Trip

Place your WebSocket relay as close to your users as possible and measure the path rather than assuming it. A simple event, like clearing the input buffer, is enough to time a round trip:

import time

async def measure_roundtrip(ws):
    """Measures network latency to the Realtime API."""
    start = time.monotonic()
    await ws.send(json.dumps({"type": "input_audio_buffer.clear"}))
    while json.loads(await ws.recv())["type"] != "input_audio_buffer.cleared":
        pass  # ignore other events arriving in between
    roundtrip_ms = (time.monotonic() - start) * 1000
    print(f"Network latency: {roundtrip_ms:.0f} ms")
    return roundtrip_ms

Keepalive and Reconnection

import asyncio
import websockets

async def robust_connection(api_key: str, max_retries: int = 5):
    """Robust connection with automatic reconnection."""
    retry_count = 0

    while retry_count < max_retries:
        try:
            async with websockets.connect(
                REALTIME_URL,
                additional_headers={
                    "Authorization": f"Bearer {api_key}"
                },
                ping_interval=20,     # Ping every 20s
                ping_timeout=10,       # Ping timeout 10s
                close_timeout=5        # Close timeout 5s
            ) as ws:
                retry_count = 0  # Reset on successful connection
                yield ws

        except websockets.exceptions.ConnectionClosed:
            retry_count += 1
            wait = min(2 ** retry_count, 30)
            print(f"Reconnecting in {wait}s...")
            await asyncio.sleep(wait)

A reconnection opens a new session: the previous conversation is no longer there, and a session lasts 60 minutes at most anyway. So keep your own record of exchanges (transcriptions, tool results) and reinject the essentials with conversation.item.create after a reconnection.

Accelerating WebRTC Setup: WARP

On the browser or native app side, OpenAI documents WARP (WebRTC Abridged Roundtrip Protocol), which reduces WebRTC session establishment time by combining three transport optimizations: DTLS 1.3, SPED, and SNAP. These are IETF drafts, supported by recent versions of libwebrtc; you can enable those your client supports, one by one. The WebRTC with WARP guide details the configuration.

Optimizing Function Calling

Function calls add latency. Optimize their execution:

import asyncio

async def execute_functions_parallel(function_calls: list) -> list:
    """Executes function calls in parallel."""
    tasks = []
    for call in function_calls:
        task = execute_single_function(call["name"], call["arguments"])
        tasks.append(task)

    results = await asyncio.gather(*tasks, return_exceptions=True)
    return results

# Cache for frequent functions
from functools import lru_cache

@lru_cache(maxsize=100)
def get_cached_data(key: str) -> dict:
    """Caches results of frequent requests."""
    # Database or API query
    return {"data": "..."}

Latency Monitoring

Integrate a measurement system to identify bottlenecks:

import time
from dataclasses import dataclass, field

@dataclass
class LatencyMetrics:
    speech_end_time: float = 0
    response_start_time: float = 0
    first_audio_time: float = 0
    response_complete_time: float = 0
    function_call_times: list = field(default_factory=list)

    @property
    def time_to_first_audio(self) -> float:
        if self.speech_end_time and self.first_audio_time:
            return (self.first_audio_time - self.speech_end_time) * 1000
        return 0

    @property
    def total_response_time(self) -> float:
        if self.speech_end_time and self.response_complete_time:
            return (self.response_complete_time - self.speech_end_time) * 1000
        return 0

    def report(self) -> dict:
        return {
            "ttfa_ms": round(self.time_to_first_audio),
            "total_ms": round(self.total_response_time),
            "function_calls_ms": [round(t * 1000) for t in self.function_call_times]
        }

class LatencyTracker:
    def __init__(self):
        self.current = LatencyMetrics()
        self.history = []

    def on_event(self, event_type: str):
        now = time.time()

        if event_type == "input_audio_buffer.speech_stopped":
            self.current.speech_end_time = now

        elif event_type == "response.output_audio.delta":
            if not self.current.first_audio_time:
                self.current.first_audio_time = now

        elif event_type == "response.done":
            self.current.response_complete_time = now
            self.history.append(self.current.report())
            self.current = LatencyMetrics()

    def average_ttfa(self) -> float:
        if not self.history:
            return 0
        return sum(h["ttfa_ms"] for h in self.history) / len(self.history)

Key Takeaways

• Total latency adds up VAD, network, model, and tools: measure each step, OpenAI does not publish typical durations
• silence_duration_ms directly controls the wait before response; semantic_vad and its eagerness are the other lever
• Concise system instructions reduce model processing time
• Parallel execution of function calls and caching reduce application latency
• A monitoring system with TTFA (Time-to-First-Audio) is essential for continuous optimization