Multimodal Voice Assistant
On this page
Updated November 2, 2026 — Audio models. Several models in this course are deprecated:
tts-1,tts-1-hd, andgpt-4o-mini-ttswill stop working on January 6, 2027 (the indicated replacement isgpt-realtime-2.1-mini, which the OpenAI TTS guide states is not supported by/v1/audio/speech: speech synthesis migration must go through the Realtime API);gpt-realtime,gpt-4o-realtime, and legacy audio models on January 20, 2027 (gpt-realtime-2.1,gpt-realtime-2.1-mini,gpt-audio-1.5);whisper-1,gpt-4o-transcribe, and/v1/audio/translationson February 26, 2027 (gpt-transcribe,gpt-live-transcribe,v1/realtime/translations). For full-duplex voice, GPT-Live 1 has been available in the API since September 10, 2026.
Corrected November 3, 2026. To analyze an image, the lesson previously made a separate call to the Responses API, then injected the description as an assistant message with
input_textcontent, a type reserved for user or system messages. However,gpt-realtime-2.1andgpt-realtime-2accept images as input: the image is attached directly, asinput_image, to a user message. Theresponse.text.deltaevent from the old beta is now namedresponse.output_text.delta. Event shapes verified againstopenaiSDK types 3.42.1. Sources: Realtime conversations, Image inputs · gpt-realtime-2.1.
Multimodal Voice Assistant
A multimodal voice assistant combines voice with other modalities: text, images, video, and structured data. The user speaks, but the agent can also display visual information, analyze images sent by the user, or combine multiple data sources. This is the future of human-computer interaction.
Text and Audio in Parallel
The Realtime API natively supports text and audio modalities simultaneously. You can inject text into the conversation while maintaining the voice exchange:
async def inject_text_context(ws, context: str):
"""Adds text context to the voice conversation."""
await ws.send(json.dumps({
"type": "conversation.item.create",
"item": {
"type": "message",
"role": "user",
"content": [
{
"type": "input_text",
"text": context
}
]
}
}))
# Trigger a voice response based on this context
await ws.send(json.dumps({"type": "response.create"}))
# The user verbally asks "Read me my latest email"
# You inject the email content as text
await inject_text_context(ws, """
Here is the latest email received:
From: [email protected]
Subject: Quote Request
Content: Hello, I would like to request a quote for
integrating a voice agent into our call center.
""")Showing an Image to the Voice Agent
gpt-realtime-2.1 and gpt-realtime-2 accept text, audio, and images as input. An image is attached as an input_image part of a user message, using a Base64 Data URI, over both WebSocket and the WebRTC data channel. The model accounts for this in its voice response, without calling another API:
import base64
import json
async def montrer_image(ws, chemin_image: str, question: str) -> None:
"""Attaches an image and a question to the conversation, then requests a voice response."""
with open(chemin_image, "rb") as f:
image_b64 = base64.b64encode(f.read()).decode()
await ws.send(json.dumps({
"type": "conversation.item.create",
"item": {
"type": "message",
"role": "user",
"content": [
{"type": "input_image", "image_url": f"data:image/jpeg;base64,{image_b64}"},
{"type": "input_text", "text": question},
],
},
}))
await ws.send(json.dumps({"type": "response.create"}))
await montrer_image(ws, "facture.jpg", "What do I need to pay, and by what date?")Images are billed as image tokens, at a specific rate listed on the model card. For heavy analysis or a more capable vision model, you can still query the Responses API separately, then pass its result to the session: do this via a system or user message (with input_text content), never via a fake assistant message, which the model would interpret as its own words.
Multimodal Web Interface
For a web application, combine the Realtime WebSocket with a visual interface:
from fastapi import FastAPI, WebSocket
from fastapi.staticfiles import StaticFiles
app = FastAPI()
@app.websocket("/ws/agent")
async def multimodal_agent(client_ws: WebSocket):
"""Multimodal agent with web interface."""
await client_ws.accept()
headers = {
"Authorization": f"Bearer {OPENAI_API_KEY}"
}
async with websockets.connect(REALTIME_URL, additional_headers=headers) as openai_ws:
await openai_ws.recv() # session.created
await openai_ws.send(json.dumps({
"type": "session.update",
"session": {
"type": "realtime",
"output_modalities": ["audio"],
"instructions": "Multimodal assistant. Respond in English.",
"audio": {"output": {"voice": "marin"}}
}
}))
await openai_ws.recv()
async def client_to_openai():
async for message in client_ws.iter_text():
data = json.loads(message)
if data["type"] == "audio":
# Relay audio
await openai_ws.send(json.dumps({
"type": "input_audio_buffer.append",
"audio": data["audio"]
}))
elif data["type"] == "text":
# Text message (chat)
await openai_ws.send(json.dumps({
"type": "conversation.item.create",
"item": {
"type": "message",
"role": "user",
"content": [{
"type": "input_text",
"text": data["text"]
}]
}
}))
await openai_ws.send(json.dumps({
"type": "response.create"
}))
async def openai_to_client():
async for message in openai_ws:
event = json.loads(message)
t = event["type"]
if t == "response.output_audio.delta":
await client_ws.send_json({
"type": "audio",
"delta": event["delta"]
})
elif t == "response.output_audio_transcript.delta":
await client_ws.send_json({
"type": "transcript",
"delta": event["delta"]
})
elif t == "response.output_text.delta":
await client_ws.send_json({
"type": "text",
"delta": event["delta"]
})
await asyncio.gather(client_to_openai(), openai_to_client())Multimodal Use Cases
Here is how a multimodal agent handles different scenarios in a function calling tool:
tools = [
{
"type": "function",
"name": "show_chart",
"description": "Displays a data chart to the user",
"parameters": {
"type": "object",
"properties": {
"chart_type": {"type": "string", "enum": ["bar", "line", "pie"]},
"title": {"type": "string"},
"data": {"type": "object"}
},
"required": ["chart_type", "title", "data"]
}
},
{
"type": "function",
"name": "send_document",
"description": "Sends a document to the client by email",
"parameters": {
"type": "object",
"properties": {
"document_type": {"type": "string"},
"recipient": {"type": "string"}
},
"required": ["document_type", "recipient"]
}
}
]The user says verbally: “Show me last quarter’s sales as a chart.” The agent calls show_chart, your frontend displays the chart, and the agent verbally comments on the results.
Key Takeaways
- The Realtime API supports text and audio modalities simultaneously in the same session
- Injecting text via
conversation.item.createallows you to enrich the voice context - Realtime models accept images: attach them as
input_imageto a user message - Tools (function calling) allow you to display visual elements while responding verbally
- The client-side WebSocket architecture allows you to mix audio, text, and structured data