Predicted Outputs: Faster Responses

Lesson updated November 3, 2026 based on the Predicted Outputs guide and OpenAI’s deprecation page: this feature exists only in Chat Completions (the prediction parameter is not accepted by the Responses API, so the previous version of this lesson would have failed) and only with models GPT-4o, GPT-4o mini, GPT-4.1, and GPT-4.1 mini, which OpenAI will retire on January 14, 2027 (GPT-4.1 nano was retired on October 23). The guide lists no newer compatible models: do not build new products on this feature without an exit plan. Sources: Predicted Outputs · Deprecations.

Speed up responses when you already know the structure

Predicted Outputs address a very common production scenario: you ask the model to modify existing content, and you know in advance that most of the result will be identical to the input. Without prediction, the model regenerates every token from scratch, including lines no one asked it to touch. With prediction, you provide the expected text; the model validates what matches and only genuinely generates the parts that differ. Latency drops accordingly.

This shift especially changes the profile of editing tasks. An assistant that rewrites a two-hundred-line file to change a function signature no longer pays for generating the entire file, but only for the touched lines: most of the work reduces to a comparison. The relevant cases all look alike: code editing, where you send the full file for a targeted change; partial rewriting, where only a single paragraph of a long document needs rephrasing; template completion, where only variable fields change; and structured data updates, where two or three values in a JSON evolve while the rest of the structure remains. In each of these cases, the expected response already exists with only minor differences.

A code editing example

The call is built like a standard Chat Completions call, augmented with a prediction parameter carrying the content you anticipate.

import openai

client = openai.OpenAI()

code_original = """
def calculate_price(quantity, unit_price, vat=0.20):
    subtotal = quantity * unit_price
    total = subtotal * (1 + vat)
    return total

def format_invoice(client, items):
    lines = []
    for item in items:
        price = calculate_price(item["qty"], item["price"])
        lines.append(f"{item['name']}: {price:.2f}€")
    return "\n".join(lines)
"""

# We predict that most of the code remains identical
instruction = (
    "Modify the calculate_price function to add a discount parameter "
    "with a default value of 0. Apply the discount before VAT. "
    "Respond with code only."
)

completion = client.chat.completions.create(
    model="gpt-4.1",
    messages=[
        {"role": "user", "content": instruction},
        {"role": "user", "content": code_original},
    ],
    prediction={
        "type": "content",
        "content": code_original,  # The prediction
    },
)

print(completion.choices[0].message.content)
# The model regenerates only the modified lines — reduced latency

The format_invoice function is not affected by the request: it will be validated as is, with no generation cost. Only the lines of calculate_price that accommodate the new parameter will be genuinely produced.

The validation mechanism

Understanding what happens under the hood avoids two common misunderstandings. The model compares its own generation, token by token, with the prediction you provided; as long as the two match, tokens are validated instantly, bypassing the full generation calculation. As soon as they diverge, normal generation takes over. The final result therefore remains correct: prediction speeds things up, it does not force anything. If your prediction is wrong, you still get the right answer, simply without the gain.

The second misunderstanding concerns price. Rejected prediction tokens are billed, which makes the operation counterproductive when the expected response has nothing to do with the prediction. Therefore, measure the acceptance rate before generalizing the pattern.

completion = client.chat.completions.create(
    model="gpt-4.1",
    messages=[
        {"role": "user", "content": "Add a discount parameter to the calculate_price function."},
        {"role": "user", "content": code_original},
    ],
    prediction={
        "type": "content",
        "content": code_original,
    },
)

usage = completion.usage
print(f"Output tokens: {usage.completion_tokens}")
print(f"Accepted tokens (prediction): "
      f"{usage.completion_tokens_details.accepted_prediction_tokens}")
print(f"Rejected tokens (prediction): "
      f"{usage.completion_tokens_details.rejected_prediction_tokens}")

# The higher the accepted/total ratio, the greater the gain

Deciding automatically

Since the gain depends entirely on the type of request, the decision to enable prediction or not should be codified rather than left to each developer. The heuristic below refuses prediction on texts that are too short, where the overhead of sending it cancels out the benefit, and reserves it for instructions that express a modification.

def should_use_prediction(original_text: str, instruction: str) -> bool:
    """Heuristic to decide if Predicted Outputs are worth it."""
    # The text must be long enough
    if len(original_text) < 200:
        return False

    # The instruction must be a targeted modification
    edit_words = ["modify", "change", "replace", "add", "fix"]
    is_edit = any(word in instruction.lower() for word in edit_words)

    return is_edit

The same reasoning applies to structured data, where the yield is even better: a configuration JSON keeps its keys, indentation, and order, and only the values move.

import json

original_data = {
    "company": "Corsen AI",
    "employees": 42,
    "departments": ["Engineering", "Sales", "Marketing"],
    "address": {
        "street": "123 Avenue de la République",
        "city": "Paris",
        "zip_code": "75011",
    },
}

json_original = json.dumps(original_data, indent=2, ensure_ascii=False)

completion = client.chat.completions.create(
    model="gpt-4.1",
    messages=[
        {"role": "user", "content": (
            "Update the following JSON: change the number of employees "
            "to 58 and add the Data Science department. Respond with JSON only."
        )},
        {"role": "user", "content": json_original},
    ],
    prediction={
        "type": "content",
        "content": json_original,
    },
)

Three caveats before deploying. Streaming is compatible, and that is even where the latency gain is most noticeable. Several parameters, however, are not: n greater than 1, logprobs, positive presence_penalty or frequency_penalty, audio, modalities other than text, max_completion_tokens, and tools (function calling). Finally, the feature is tied to end-of-life models: GPT-4o, GPT-4o mini, GPT-4.1, and GPT-4.1 mini stop on January 14, 2027, and no newer model is announced as compatible. For fast editing on current models, the sustainable path is to ask the model for a diff or a patch rather than the entire file.

Key takeaways

• Predicted Outputs speed up responses when you know the expected structure — in Chat Completions only
• Ideal for code editing, JSON updates, partial rewriting
• Check accepted_prediction_tokens to measure the real gain
• Rejected prediction tokens are billed: do not predict at random
• Do not use them for entirely new generations
• Compatible models (GPT-4o, GPT-4.1 and their mini versions) retired on January 14, 2027: plan an alternative, such as diff generation