Streaming pipeline from LLM to TTS

Updated November 3, 2026, based on the xAI Text to Speech page: each text.delta can hold up to 60,000 characters. Source: Text to Speech.

Connect an LLM to real-time speech synthesis

The most powerful use of TTS WebSocket is chaining it to a streaming LLM. The LLM generates text token by token, each token goes to TTS over WebSocket, and the audio plays as it arrives. The user hears the answer while the LLM is still thinking.

Pipeline architecture

The data flows through three stages:

User -> LLM (streaming) -> TTS WebSocket -> Audio (playback)
question   text tokens        audio.delta       speaker

Every stage streams, which minimizes perceived latency. The user doesn’t wait for the LLM to finish generating its whole answer before starting to hear it.

Implementation in Python

Here’s a complete pipeline that connects the Grok Chat Completions API to TTS WebSocket:

import asyncio
import websockets
import json
import base64
import aiohttp

async def llm_to_speech(api_key, prompt, voice="rex", language="en"):
    # Step 1: TTS WebSocket connection
    tts_uri = f"wss://api.x.ai/v1/tts?language={language}&voice={voice}"
    tts_headers = {"Authorization": f"Bearer {api_key}"}

    audio_chunks = []

    async with websockets.connect(tts_uri, additional_headers=tts_headers) as tts_ws:

        # Step 2: Streaming LLM call
        async with aiohttp.ClientSession() as session:
            async with session.post(
                "https://api.x.ai/v1/chat/completions",
                headers={
                    "Authorization": f"Bearer {api_key}",
                    "Content-Type": "application/json"
                },
                json={
                    "model": "grok-4.5",
                    "messages": [{"role": "user", "content": prompt}],
                    "stream": True
                }
            ) as llm_response:

                # Step 3: Each LLM token -> TTS delta
                buffer = ""
                async for line in llm_response.content:
                    line = line.decode().strip()
                    if not line.startswith("data: "):
                        continue
                    if line == "data: [DONE]":
                        break

                    chunk = json.loads(line[6:])
                    token = chunk["choices"][0].get("delta", {}).get("content", "")

                    if token:
                        buffer += token
                        # Send complete sentences
                        if any(c in token for c in ".!?\n"):
                            await tts_ws.send(json.dumps({
                                "type": "text.delta",
                                "delta": buffer
                            }))
                            buffer = ""

                # Send whatever is left in the buffer
                if buffer:
                    await tts_ws.send(json.dumps({
                        "type": "text.delta",
                        "delta": buffer
                    }))

                await tts_ws.send(json.dumps({"type": "text.done"}))

        # Step 4: Collect the audio
        async for msg in tts_ws:
            data = json.loads(msg)
            if data["type"] == "audio.delta":
                audio_chunks.append(base64.b64decode(data["delta"]))
            elif data["type"] == "audio.done":
                break

    return b"".join(audio_chunks)

Buffering strategy

Sending each token to TTS individually isn’t ideal: the synthesis engine produces better results when it has context. There are three buffering strategies.

By sentence

Wait for a sentence delimiter (., !, ?, line break) before sending:

if any(c in token for c in ".!?\n"):
    await tts_ws.send(json.dumps({
        "type": "text.delta", "delta": buffer
    }))
    buffer = ""

This is the best balance between quality and latency.

By size

Send when the buffer reaches a certain size (200 characters, for example):

if len(buffer) >= 200:
    await tts_ws.send(json.dumps({
        "type": "text.delta", "delta": buffer
    }))
    buffer = ""

It’s simpler but can cut a word in half.

By timeout

Send after a delay with no new token (useful while the LLM is thinking):

last_token_time = time.time()

# If more than 500 ms pass without a token, send the buffer
if time.time() - last_token_time > 0.5 and buffer:
    await tts_ws.send(...)
    buffer = ""

Real-time audio playback

To play the audio as it arrives (not just collect it), use an async queue:

import asyncio

audio_queue = asyncio.Queue()

async def receive_audio(tts_ws):
    """Receives audio chunks and puts them in the queue."""
    async for msg in tts_ws:
        data = json.loads(msg)
        if data["type"] == "audio.delta":
            audio_bytes = base64.b64decode(data["delta"])
            await audio_queue.put(audio_bytes)
        elif data["type"] == "audio.done":
            await audio_queue.put(None)  # End sentinel
            break

async def play_audio():
    """Plays audio chunks from the queue."""
    while True:
        chunk = await audio_queue.get()
        if chunk is None:
            break
        # Play the chunk (pyaudio, sounddevice, etc.)
        play_chunk(chunk)

By running receive_audio and play_audio in parallel, playback starts with the first chunk received.

Error handling in the pipeline

The pipeline can fail in three places: the LLM, TTS, or the network. Handle each one:

async def robust_pipeline(api_key, prompt):
    try:
        audio = await llm_to_speech(api_key, prompt)
        return audio
    except aiohttp.ClientError:
        print("LLM error: check your connection")
    except websockets.ConnectionClosed:
        print("TTS connection lost: reconnection needed")
    except json.JSONDecodeError:
        print("Invalid response: unexpected format")
    except Exception as e:
        print(f"Unexpected error: {e}")
    return None

Key takeaways

• A streaming LLM feeding TTS WebSocket gives the lowest latency
• Buffer by sentence for the best balance between quality and latency
• Use an async queue for real-time audio playback
• Handle errors at every stage of the pipeline
• Total text over WebSocket is unlimited (deltas of up to 60,000 characters)
• The limit of 50 concurrent sessions applies to your whole team