Real-time streaming

Course updated November 3, 2026 for Voxtral TTS 3, based on the Migrate from Voxtral 2 guide and Mistral’s Text to Speech pages: voxtral-mini-tts-2603, deprecated October 31, 2026 and retired April 30, 2027, is replaced in examples by voxtral-mini-tts-3-0. Voxtral 2 predefined voices (gb_jane_neutral…) remain supported; Voxtral 3 adds [emotion=…] and [pace=…] tags in text, the expressiveness setting, custom pronunciation (phoneme_hints), and 35 languages. Mistral recommends re-measuring latency after migration. Sources: Migrate from Voxtral 2 · Speech generation.

Why streaming changes everything

So far, you’ve worked in synchronous mode: you send the text, wait for synthesis to finish completely, then receive the full audio file. For a pre-generated greeting created the day before, this wait doesn’t matter. For a voice assistant answering a customer on the phone, it’s a dealbreaker—two seconds of silence after a question, and the user thinks the call dropped.

Streaming eliminates this wait. Audio ships as it’s generated, fragment by fragment, so the user hears the first words while the model is still producing the end of the sentence. Total generation time doesn’t change; what collapses is the delay before the first sound.

The event protocol

The stream relies on Server-Sent Events. Each message received is a typed event, and two types concern you:

• speech.audio.delta: contains an audio fragment in base64. This is the main event you’ll handle
• speech.audio.done: end signal—generation is complete

On the code side, switching to streaming means adding stream=True and consuming the response in a with block. The following example accumulates fragments in memory to write a file at the end: this isn’t real-time yet, but it’s the loop structure the rest of the lesson will build on.

import base64
from pathlib import Path
from mistralai.client import Mistral

client = Mistral(api_key="your-api-key")

audio_chunks = []

with client.audio.speech.complete(
    model="voxtral-mini-tts-3-0",
    input="Streaming makes voice agents responsive and natural.",
    voice_id="your-voice-id",
    response_format="opus",
    stream=True,
) as stream:
    for event in stream:
        if event.event == "speech.audio.delta":
            chunk = base64.b64decode(event.data.audio_data)
            audio_chunks.append(chunk)
            print(f"Chunk received: {len(chunk)} bytes")

# Assemble all chunks
audio_complet = b"".join(audio_chunks)
Path("streaming-output.opus").write_bytes(audio_complet)
print(f"Final audio: {len(audio_complet)} bytes")

Playing sound as it arrives

To get the promised responsiveness, stop accumulating and write each fragment directly to the sound card. With pyaudio, the change takes one line: instead of adding the decoded fragment to a list, you send it to an output stream opened upfront, configured for mono Float32 at 24,000 Hz.

import base64
import pyaudio
from mistralai.client import Mistral

client = Mistral(api_key="your-api-key")

# PCM audio configuration (fastest format)
p = pyaudio.PyAudio()
stream_audio = p.open(
    format=pyaudio.paFloat32,
    channels=1,
    rate=24000,
    output=True,
)

with client.audio.speech.complete(
    model="voxtral-mini-tts-3-0",
    input="You're hearing this sentence while I generate it.",
    voice_id="your-voice-id",
    response_format="pcm",
    stream=True,
) as stream:
    for event in stream:
        if event.event == "speech.audio.delta":
            audio_data = base64.b64decode(event.data.audio_data)
            stream_audio.write(audio_data)

stream_audio.stop_stream()
stream_audio.close()
p.terminate()

The requested format changed from opus to pcm, and that matters. PCM needs no decoding: received bytes are ready to play immediately, making it the recommended format for real-time use.

Relaying the stream to a browser

In a web app, the server doesn’t play sound: it relays it. The classic pattern uses a WebSocket that receives text from the browser, opens the Voxtral stream, and pushes each decoded fragment back to the client. On the browser side, the Web Audio API takes over to play received PCM fragments.

# Server side (FastAPI example)
from fastapi import FastAPI, WebSocket
import base64
from mistralai.client import Mistral

app = FastAPI()
client = Mistral(api_key="your-api-key")

@app.websocket("/ws/tts")
async def tts_websocket(websocket: WebSocket):
    await websocket.accept()
    texte = await websocket.receive_text()

    with client.audio.speech.complete(
        model="voxtral-mini-tts-3-0",
        input=texte,
        voice_id="your-voice-id",
        response_format="pcm",
        stream=True,
    ) as stream:
        for event in stream:
            if event.event == "speech.audio.delta":
                await websocket.send_bytes(
                    base64.b64decode(event.data.audio_data)
                )

    await websocket.close()

The setting that makes the difference: the buffer

Fragment sizes vary and typically represent a few milliseconds of audio each. If you play the first fragment the moment it arrives, any network irregularity causes an audible micro-dropout—a hiccup in the voice that immediately gives away the machine. The fix is to buffer two or three fragments before starting playback, costing a few tens of milliseconds and smoothing the rest of the stream.

Then there’s the format choice, which trades off latency against bandwidth. PCM delivers the first sound fastest—around 0.7 seconds in measurements done with Voxtral 2, to be redone with Voxtral TTS 3—at the cost of high throughput. Opus loses a bit on latency but compresses heavily, making it the right compromise when your users are on mobile.

Key takeaways

• Streaming sends audio chunk by chunk during generation
• Enable it with stream=True in the audio.speech.complete() call
• speech.audio.delta events contain audio fragments in base64
• PCM format offers the lowest latency for streaming (~0.7s vs ~2s for MP3)
• For real-time playback, use pyaudio or WebSockets + Web Audio API