Reduce Latency with Predicted Outputs

The Latency Problem in Production

In production, every millisecond counts. When a user waits for an LLM response to make a minor code or text edit, standard token-by-token generation latency can feel excessive. If 90% of the expected output is already known, why should the model regenerate it entirely?

Predicted Outputs solve this problem. By providing the model with a prediction of the output, you let it allocate compute resources only to the parts that actually differ, drastically reducing response time.

The Concept: Predict to Accelerate

The principle is simple. You already know most of the expected response—for example, an existing code file you want to modify slightly. Instead of letting the model regenerate everything, you pass the expected content via the prediction parameter:

response = client.chat.complete(
    model="mistral-large-latest",
    messages=[...],
    prediction={
        "type": "content",
        "content": expected_content
    }
)

The model then uses this content as a baseline. It dedicates its compute resources to tokens that differ from the prediction, which significantly speeds up generation.

How It Works Technically

The mechanism relies on accelerated validation rather than shortened generation. The model receives your prediction as working material, then proceeds with generation by comparing its output to the predicted content. Whenever the token it is about to produce matches the prediction, validation is nearly instantaneous—the full generation compute does not occur. As soon as its output diverges from the prediction, it resumes normal token-by-token generation until it potentially realigns with the predicted content.

The practical outcome is twofold. First, response quality remains strictly preserved: the model does not “copy” your prediction; it validates or deviates from it based on its own judgment. Second, the time savings are proportional to the overlap rate—a prediction that is 90% accurate speeds things up significantly, while a prediction that is only 40% accurate offers almost no benefit.

Compatible Models

As of April 2026, two models support Predicted Outputs: mistral-large-latest for complex editing and reasoning tasks, and codestral-latest, optimized for code editing. The latter accounts for most real-world use cases, since code editing is the ideal scenario for prediction.

When to Use Predicted Outputs

The criterion boils down to one question: do you already know most of the output? When you ask Codestral to rename a variable in a 400-line file, the expected response is that same file with only a few occurrences changed—pass the original file as the prediction, and generation becomes nearly instant. The same logic applies to correcting a sentence in a long document, filling a template with a fixed structure, or modifying a key in a large JSON file: in all these cases, more than 90% of the output is known in advance, and that is precisely what prediction leverages.

Conversely, the feature loses all value when the output is unpredictable—creative responses, brainstorming, or open-ended summaries. If the actual content diverges from the prediction for more than half the tokens, the benefit vanishes, and the model spends its time noting discrepancies. Also note a technical limitation: Predicted Outputs do not work with the n parameter (multiple generations).

Impact on Pricing

Good news: Predicted Outputs have no impact on pricing. You do not pay extra to provide a prediction. You pay only for the tokens actually generated, as usual.

Key Takeaways

• Predicted Outputs reduce latency by leveraging predictable content
• The prediction parameter provides the model with the expected output as a baseline
• Two compatible models: mistral-large-latest and codestral-latest
• Ideal for code editing, text modification, and templates
• No extra cost—quality is preserved