Prompt caching: Reuse computations

What is prompt caching?

When you repeatedly send the same prompt prefix—a long system prompt, detailed instructions, or a reference document—OpenAI can automatically cache that prefix. Subsequent calls that share this prefix benefit from reduced time to first token (TTFT) and lower costs on input tokens. In other words, the work of ingesting the stable context isn’t repeated for every request.

Prompt caching has been automatic since 2025: you don’t need to enable anything. OpenAI detects identical prefixes and caches them. Your leverage, therefore, isn’t turning it on or off, but writing your prompts so the mechanism can apply. This is a matter of structure, not configuration, and it’s where most teams leave money on the table.

How it works

The cache applies to prompt prefixes: the first tokens must be strictly identical from one call to the next, and the match stops at the first divergent character. A minimum size of 1,024 tokens is required to trigger caching, which immediately rules out short system prompts. The cache is shared within a single organization, meaning multiple services in your company calling the same prefix help each other. Its lifespan ranges from a few minutes to a few hours, with no firm guarantee, and the reduction achieved is approximately 50% on the cost of cached input tokens.

The practical consequence of the prefix rule is that you must place stable elements at the start of the prompt and variable elements at the end. A service that inserts today’s date or a session ID at the head of its system prompt invalidates its cache on every call without ever realizing it.

import openai

client = openai.OpenAI()

# ✅ Good structure: stable elements first
SYSTEM_PROMPT = """You are a legal assistant specializing in French
contract law. You analyze contractual clauses according to the
Civil Code and recent case law. You always respond by
structuring your analysis: 1) Clause identification,
2) Legal analysis, 3) Potential risks, 4) Recommendations.

Here is the reference contract to analyze:
[Insert a 2,000+ token document here that does not change between calls]
"""

def analyze_clause(clause: str) -> str:
    """Each call reuses the system prompt cache."""
    response = client.responses.create(
        model="gpt-5.6-terra",
        instructions=SYSTEM_PROMPT,  # Stable prefix = cached
        input=f"Analyze this clause: {clause}",  # Variable part
    )
    return response.output_text

In this example, the reference contract and the analysis framework constitute the invariant block, while the clause being examined changes with each call. Analyze forty clauses from the same contract, and you pay for the context once at full price.

Verifying the cache

An optimization you don’t measure is an optimization you only think you have. The API returns information about cache usage in the response:

response = client.responses.create(
    model="gpt-5.6-terra",
    instructions=SYSTEM_PROMPT,
    input="Analyze the non-compete clause.",
)

# Check cache usage
usage = response.usage
print(f"Input tokens: {usage.input_tokens}")
print(f"Input tokens (cached): {usage.input_tokens_details.cached_tokens}")
print(f"Output tokens: {usage.output_tokens}")

# If cached_tokens > 0: the cache was used
cache_ratio = usage.input_tokens_details.cached_tokens / usage.input_tokens
print(f"Cache ratio: {cache_ratio:.1%}")

Log this ratio in production rather than checking it once in development. A sudden drop in the cache rate is the most reliable symptom of an unintentional change in prompt construction, and you’ll see it before the monthly bill tells you about it.

Structuring to maximize cache

The first approach is to deliberately enrich the system prompt. This feels counterintuitive when you’re trying to reduce tokens, but it aligns with the threshold rule: a 900-token system prompt will never be cached, whereas the same prompt enriched with few-shot examples and a reference document crosses the line and switches to the reduced rate for the cached portion.

# Enrich the system prompt to maximize cache
ENRICHED_PROMPT = """
[Detailed instructions - 500+ tokens]
[Few-shot examples - 500+ tokens]
[Reference document - 1,000+ tokens]
---
Total > 1,024 tokens = eligible for cache
"""

In a multi-turn conversation, the mechanism works in your favor without extra effort: the accumulated history forms a naturally stable prefix that grows with each exchange. As long as you always reinject messages in the same order, turn number ten benefits from the cache built by the previous nine.

def conversation_with_cache(history: list[dict], new_message: str):
    """Growing history maximizes cache automatically."""
    history.append({"role": "user", "content": new_message})

    response = client.responses.create(
        model="gpt-5.6-terra",
        instructions=SYSTEM_PROMPT,
        input=history,
    )

    history.append({
        "role": "assistant",
        "content": response.output_text,
    })
    return response.output_text

Finally, since cache lifespan is measured in minutes, grouping requests that share a prefix becomes a scheduling decision. Processing forty clauses from a contract back-to-back keeps the cache hot; spreading them out over the day while interleaving other files lets it expire between calls.

import asyncio

async def batch_with_cache(clauses: list[str]) -> list[str]:
    """Sends requests quickly to leverage the cache."""
    results = []
    for clause in clauses:
        result = analyze_clause(clause)
        results.append(result)
    return results

Three mistakes cancel out all this work, and they look alike: modifying the start of the prompt, even by a single character, invalidates the cache for the entire request; randomizing the order of instructions, for example by building the prompt from an unordered dictionary, produces the same effect intermittently and is therefore hard to diagnose; and expecting gains below the 1,024-token threshold means trying to optimize something that never triggers.

Key takeaways

• Prompt caching is automatic with OpenAI—place stable elements at the start of the prompt
• Minimum threshold: 1,024 identical tokens in the prefix
• Gain: ~50% on cached input tokens + reduced TTFT
• Check usage.input_tokens_details.cached_tokens to confirm caching