Streaming SSE

Real-time responses with Server-Sent Events

By default, the Chat Completions API waits until the model has finished generating its entire response before sending it back to you. For a short question, nobody notices the delay. For a thousand-word report, however, it means several seconds of a blank screen, leaving your user wondering if the application has crashed. Streaming solves this problem by sending tokens as they are generated, creating the familiar typewriter effect seen in conversational interfaces.

The protocol used is Server-Sent Events (SSE), an HTTP standard that allows the server to push data to the client continuously over a persistent connection. There is nothing exotic here: no WebSockets to install, no extra dependencies.

Streaming vs. non-streaming

In non-streaming mode, which is the default, the client sends the request and waits for the complete response:

Client ──── request ────► Server
         (wait 3-10s)
Client ◄── complete response ── Server

The advantage is simplicity: the response is atomic, a single JSON object that is easy to process, validate, and store. The downside is just as clear: perceived latency is high because the user sees nothing during the entire generation process.

In streaming mode, the server sends tokens one by one, or in small groups:

Client ──── request ────► Server
Client ◄── chunk 1 ─────── Server  (after ~200ms)
Client ◄── chunk 2 ─────── Server  (after ~250ms)
Client ◄── chunk 3 ─────── Server  (after ~300ms)
...
Client ◄── [DONE] ──────── Server

Perceived latency drops dramatically—the first character appears within a few hundred milliseconds—significantly improving the user experience. In return, you must reconstruct the message on the client side and manage state that no longer arrives as a single block.

The SSE protocol in detail

Here is what the HTTP stream you receive actually looks like:

data: {"id":"cmpl-abc","choices":[{"delta":{"content":"Hello"}}],"usage":null}

data: {"id":"cmpl-abc","choices":[{"delta":{"content":" "}}],"usage":null}

data: {"id":"cmpl-abc","choices":[{"delta":{"content":"!"}}],"usage":null}

data: [DONE]

Each line starts with data: followed by a JSON object, and empty lines serve as separators between events. Note the detail that traps everyone on their first implementation: the text fragment is located in the delta field, not in message as in classic mode. The stream ends with a data: [DONE] event, which explicitly signals the end. Without it, you would not be able to distinguish a finished generation from an interrupted connection.

Enable streaming in Python

With the Mistral SDK, you enable streaming by calling chat.stream() instead of chat.complete():

from mistralai.client import Mistral
import os

client = Mistral(api_key=os.getenv("MISTRAL_API_KEY"))

stream = client.chat.stream(
    model="mistral-large-latest",
    messages=[
        {"role": "user", "content": "Explain how an electric motor works."}
    ]
)

for chunk in stream:
    content = chunk.data.choices[0].delta.content
    if content:
        print(content, end="", flush=True)

print()  # Final newline

The flush=True argument deserves an explanation, as forgetting it is the classic cause of streaming that “doesn’t work.” Without it, Python keeps characters in the buffer and displays them in blocks. This visually cancels out all the benefits of streaming, even though the chunks are arriving correctly.

Structure of a chunk

A streaming chunk does not read like a complete response, and the access paths differ:

# Non-streaming: response.choices[0].message.content
# Streaming:     chunk.data.choices[0].delta.content

# The first chunk often contains the role
# chunk.data.choices[0].delta.role == "assistant"

# Subsequent chunks contain text fragments
# chunk.data.choices[0].delta.content == "Hello"

# The last chunk contains finish_reason
# chunk.data.choices[0].finish_reason == "stop"

The first chunk usually carries the role without text, subsequent ones carry the fragments, and the last one provides the finish_reason. This is why the if content: check in the previous code is not an unnecessary precaution: concatenating a None value would cause the loop to fail on the first iteration.

Reconstructing the complete response

Displaying fragments is not enough when you need to keep the response—for example, to add it to the conversation history, log it, or store it in a database. You must therefore accumulate the text in parallel with displaying it:

from mistralai.client import Mistral
import os

client = Mistral(api_key=os.getenv("MISTRAL_API_KEY"))

stream = client.chat.stream(
    model="mistral-large-latest",
    messages=[{"role": "user", "content": "List 5 programming languages."}]
)

full_response = ""
finish_reason = None

for chunk in stream:
    delta = chunk.data.choices[0].delta
    if delta.content:
        full_response += delta.content
        print(delta.content, end="", flush=True)

    if chunk.data.choices[0].finish_reason:
        finish_reason = chunk.data.choices[0].finish_reason

print(f"\n\nFinish reason: {finish_reason}")
print(f"Complete response ({len(full_response)} characters)")

The finish_reason variable is captured in the same loop for the same reason as in classic mode: a response truncated by max_tokens appears on screen exactly like a complete response, and only this field will tell you the difference.

When to use streaming

Streaming is not always better. It is indicated whenever a human is watching the screen, but superfluous or even detrimental when a machine is processing the output.

The last case is the most instructive. If you are expecting JSON, streaming adds nothing: you cannot parse an incomplete object, and you will have to wait for the last fragment anyway. You might as well keep the call atomic and the code simple.

Key takeaways

• Streaming reduces perceived latency by sending tokens as they are generated
• The SSE protocol is a simple and robust HTTP standard
• In streaming mode, use delta.content instead of message.content
• Accumulate fragments on the client side to reconstruct the complete response
• Prefer streaming for user interfaces, and non-streaming for automated processing