Performance in Production

Beyond Cache: Overall Performance

Cache cuts costs, but production integration depends on more than just the bill: user-perceived latency, sustained throughput, reliability against transient errors, and the ability to diagnose degradation. An application that costs little but responds in 12 seconds or fails one in 50 times isn’t performant. This lesson covers the four dimensions that matter once you’ve mastered cost.

Streaming: Saving Time Without Actually Saving Time

For any interface where a human waits for a response, SSE (Server-Sent Events) streaming changes the perception radically. Without it, the user stares at a cursor while the model generates 500 tokens; with it, the first words appear after a few hundred milliseconds and reading begins during generation. The total time is the same, but the perceived time is cut by a factor of several.

import requests
import json

def appel_streaming(messages, model="grok-4.20"):
    reponse = requests.post(
        "https://api.x.ai/v1/chat/completions",
        headers={
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json"
        },
        json={
            "model": model,
            "messages": messages,
            "stream": True
        },
        stream=True
    )

    for ligne in reponse.iter_lines():
        if ligne and ligne.startswith(b"data: "):
            donnees = ligne[6:]
            if donnees == b"[DONE]":
                break
            chunk = json.loads(donnees)
            token = chunk["choices"][0]["delta"].get("content", "")
            yield token

The stream: True parameter changes neither the number of billed tokens nor their rate. What you improve is the TTFT—the time to first token. Note that the function is a generator: it yields control token by token, letting you write directly to your server’s HTTP response without keeping the entire generation in memory.

Choosing Your Model for Latency

The second latency lever is model choice, and it hinges mainly on reasoning effort. A reasoning model produces invisible reasoning tokens before drafting its response: quality goes up, delay goes up too. At the extremes, grok-build-0.1 is the fastest in the catalog, making it suited for real-time uses like on-the-fly classification or autocompletion; grok-4.20-0309 in non-reasoning variant and grok-4.3 offer a reasonable compromise between quality and speed; grok-4.5 with deep reasoning sits at the other end, reserved for tasks that truly require it.

The practical consequence is that you don’t choose a model for an application, but for a step. A common pattern is to have a fast initial call decide—is this request trivial, ambiguous, or complex?—then send only the fraction of cases that justify it to a reasoning model. You divide both average latency and average cost without degrading difficult cases.

Handling Errors Without Making Things Worse

The 429 code covered in Lesson 3 isn’t the only one you’ll encounter. The key distinction lies between errors that deserve a retry and those where retrying just adds unnecessary traffic.

You can code this rule in three lines, and it’s worth isolating in a function rather than scattering if statements everywhere:

def classifier_erreur(status_code):
    recuperable = {429, 500, 502, 503}
    return status_code in recuperable

Timeouts and Resilience

A poorly set timeout creates two opposite issues: too short, and it abandons requests that would have succeeded; too long, and it blocks your workers while the queue lengthens. Set three separately. The connection must establish in 10 seconds; beyond that, the problem is network. Reading in non-streaming mode scales with the model: 120 seconds for a reasoning model, 60 seconds for grok-build-0.1. In streaming, the relevant threshold isn’t total duration but the interval between chunks, set around 30 seconds.

One point deserves real caution: when a timeout expires on a long generation, do not automatically restart. The model may have already consumed reasoning tokens that you will pay for a second time, and nothing guarantees the first request didn’t succeed server-side. An idempotence mechanism—a request key you keep to recognize a duplicate—or an explicit resume works better than a blind retry.

What to Instrument

Four metric families suffice to diagnose most incidents.

Add latency at the 95th percentile. The average always masks extreme cases, and those are what trigger support tickets: a P95 at 8 seconds means one user in 20 waits 8 seconds, which no 1.2-second average would reveal.

Key Takeaways

• SSE streaming reduces perceived latency without affecting cost
• Choose a fast model for real-time and a reasoning model for quality
• Distinguish recoverable errors (429, 500, 502/503) from final errors (400, 401)
• Configure timeouts suited to the model in use
• Monitor TTFT, success rate, cache rate, and average cost per request