WebSocket Architecture of the Realtime API

Updated November 2, 2026 — Audio models. Several models in this course are deprecated: tts-1, tts-1-hd, and gpt-4o-mini-tts will stop working on January 6, 2027 (replacement: gpt-realtime-2.1-mini, which the OpenAI TTS guide states is not supported by /v1/audio/speech: speech synthesis migration moves to the Realtime API); gpt-realtime, gpt-4o-realtime, and legacy audio models on January 20, 2027 (gpt-realtime-2.1, gpt-realtime-2.1-mini, gpt-audio-1.5); whisper-1, gpt-4o-transcribe, and /v1/audio/translations on February 26, 2027 (gpt-transcribe, gpt-live-transcribe, v1/realtime/translations). For full-duplex voice, GPT-Live 1 has been available in the API since September 10, 2026. Addendum November 3, 2026: section on OpenAI’s three voice architectures (GPT-Live, Realtime API, processing chain) and transport selection (WebRTC, WebSocket, SIP), based on the Voice agents, Audio and voice, and Getting started with the Realtime API guides; maximum session duration: 60 minutes (Realtime conversations).

OpenAI’s Realtime API represents a fundamental paradigm shift in how you interact with language models. Unlike traditional REST APIs that operate on a request-response cycle, the Realtime API uses the WebSocket protocol to establish a persistent, bidirectional connection between your application and the model.

This architecture enables real-time audio exchange: you send audio, the model processes it, and responds vocally with low latency, skipping intermediate transcription or text-to-speech steps. This is one of the core building blocks for conversational voice agents and the heart of this course.

Three Voice Architectures in 2026

Before opening a WebSocket, place the Realtime API in context alongside OpenAI’s other solutions. The Voice agents guide distinguishes three approaches:

Architecture	Best for	Why choose it
GPT-Live	Full-duplex conversations with a separate backend	The voice model listens while speaking and delegates reasoning and tool use to a backend of your choice; keeps your existing text agent intact
Realtime API	Speech, reasoning, and tools in a single session	A single model interprets audio, decides what to do, and speaks the response
Processing chain	Granular control over every step	Transcription, then text agent, then speech synthesis: you inspect or transform text between steps and swap out each component independently

For new voice conversation applications, OpenAI’s Audio and voice page now recommends starting with GPT-Live. The Realtime API remains the right starting point when the agent must fit within a single session, supporting user interruption, fast first audio response, natural turn-taking, and direct tool calls. The processing chain works best when every step must remain visible: retaining transcripts, enforcing rules before the agent replies, or emitting voice only after validating the response.

This course focuses on the Realtime API and the processing chain, whose components (transcription, speech synthesis) are covered in subsequent sections.

WebRTC, WebSocket, or SIP

Your transport choice depends on where the audio originates. According to the documentation: use WebRTC for browser audio, with an ephemeral client secret generated by your server; WebSocket for server-side audio pipelines; SIP for telephone calls. A server-side control connection also allows a trusted backend to observe and steer a media session. This lesson details WebSocket, the simplest way to understand exchanged events; in a browser, prefer WebRTC. A Realtime session lasts at most 60 minutes.

WebSocket vs. REST: Why This Choice?

In a traditional REST architecture, every interaction requires a complete HTTP request: opening a TCP connection, TLS negotiation, sending headers, waiting for the response, and closing the connection. For real-time audio, this overhead is prohibitive.

Criterion	REST (Responses API)	WebSocket (Realtime API)
Latency	200–500 ms per request	< 50 ms after connection
Direction	Client → Server only	Simultaneous bidirectional
Connection	New for every request	Persistent (session)
Audio	Full file required	Chunk-by-chunk streaming

The WebSocket protocol keeps a channel open continuously. Once established, client and server can send messages at any time without waiting for the other to finish.

Establishing a WebSocket Connection

Here’s how to establish a connection to the Realtime API in Python:

import asyncio
import websockets
import json

OPENAI_API_KEY = "sk-..."
URL = "wss://api.openai.com/v1/realtime?model=gpt-realtime-2.1"

async def connect_realtime():
    headers = {
        "Authorization": f"Bearer {OPENAI_API_KEY}"
    }

    async with websockets.connect(URL, additional_headers=headers) as ws:
        print("Connected to Realtime API")

        # Receive the welcome event
        response = await ws.recv()
        event = json.loads(response)
        print(f"Type: {event['type']}")
        # → "session.created"

        # Keep the connection open
        await asyncio.Future()

asyncio.run(connect_realtime())

The connection URL includes the model as a query parameter. Since the API reached general availability, only the authentication header is required—the OpenAI-Beta: realtime=v1 header from the beta phase is gone.

Interface noted August 5, 2026 — The Realtime API evolves (models, event names, session format): when in doubt, check the official documentation.

Realtime Application Architecture

An application using the Realtime API follows a layered architecture:

Audio layer (capture/playback)

Captures the microphone, plays audio responses. Uses PyAudio, sounddevice, or the Web Audio API in the browser.

WebSocket layer (transport)

Manages the persistent connection, encodes/decodes JSON events, handles reconnections.

Application layer (business logic)

Orchestrates the conversation, manages context, calls tools, processes results.

Server layer (optional proxy)

In production, an intermediary server protects the API key and adds server-side business logic.

The Role of the Proxy Server

During development, you can connect your application directly to the Realtime API. In production, a proxy server is essential to:

Protect your API key — never expose it on the client side
Authenticate your users — verify tokens before opening the session
Control costs — limit session duration and token usage
Log interactions — for debugging and compliance

# Simplified example of a WebSocket proxy with FastAPI
from fastapi import FastAPI, WebSocket
import websockets

app = FastAPI()

@app.websocket("/ws/realtime")
async def proxy_realtime(client_ws: WebSocket):
    await client_ws.accept()

    headers = {"Authorization": f"Bearer {OPENAI_API_KEY}"}

    async with websockets.connect(URL, additional_headers=headers) as openai_ws:
        async def forward_to_openai():
            async for message in client_ws.iter_text():
                await openai_ws.send(message)

        async def forward_to_client():
            async for message in openai_ws:
                await client_ws.send_text(message)

        await asyncio.gather(forward_to_openai(), forward_to_client())

Key Takeaways

Three voice architectures at OpenAI: GPT-Live (recommended for new conversational apps), Realtime API (speech, reasoning, and tools in one session), and the transcription → agent → synthesis chain
WebRTC for browser audio, WebSocket for server pipelines, SIP for telephony; sessions last at most 60 minutes
Realtime API uses WebSocket for low-latency bidirectional communication
The connection is persistent: one opening for the entire conversational session
Layered architecture separates audio, transport, business logic, and server proxy
In production, a proxy server is essential for security and cost control
The protocol is event-driven: every message is a JSON object with a type field