Advanced prompt caching

Beyond automatic caching

You saw the basics of prompt caching in Lesson 3. There, caching appeared as an automatic mechanism that benefits you without any special effort, and that’s true as long as your application sends well-ordered requests. Production reality is messier: multiple users hit the same API simultaneously, each with their own folder, documents, and questions. The common prefix then shrinks to almost nothing, and the cache hit rate collapses without anyone noticing, since nothing breaks—the bill just goes up.

Multi-user, multi-document, processing pipelines: we’ll tackle these complex scenarios one by one to maximize the cache hit rate. The guiding principle fits in one sentence: caching isn’t something that happens to you; it’s something you design.

Prompt architecture to maximize caching

Caching applies to the longest common prefix across your requests. The longer and more stable this prefix, the more you save. The practical consequence is that a prompt must be organized from most stable to most variable, like a book whose first pages never change and only the final chapter is rewritten with each edition.

The example below applies this principle to a document analysis assistant for a law firm. Three layers follow one another: global instructions, which are frozen; the knowledge base, revised once a month; and the case file and question, which change with every call.

import openai

client = openai.OpenAI()

# Layered architecture: from most stable to most variable

# Layer 1: Global instructions (never change)
GLOBAL_INSTRUCTIONS = """
You are a document analysis assistant for a law firm.
Rules:
- Always respond in French
- Cite relevant legal articles
- Structure as: Analysis / Risks / Recommendations
- Be factual and precise
"""

# Layer 2: Knowledge base (changes rarely)
KNOWLEDGE_BASE = """
[Insert 5,000+ tokens of case law, legal articles,
 analysis templates here—this content changes at most once a month]
"""

# Layer 3: Case context (changes per client)
def build_prompt(case_context: str, question: str) -> str:
    return f"""{GLOBAL_INSTRUCTIONS}

{KNOWLEDGE_BASE}

--- Current Case ---
{case_context}

--- Question ---
{question}"""

The classic mistake is placing the client name or today’s date at the start of the system prompt to “personalize” it. A single variable character in the initial position is enough to make the prefix unique, reducing the cache hit rate to zero for all subsequent layers. Anything that varies must go last.

Of course, you need to verify that the theory translates into numbers. The following monitor reads input_tokens_details.cached_tokens on every response and accumulates the measurements, giving you a usable average rate rather than a gut feeling.

class CacheMonitor:
    """Monitors prompt caching efficiency."""

    def __init__(self):
        self.calls: list[dict] = []

    def record(self, usage) -> dict:
        total_input = usage.input_tokens
        cached_tokens = getattr(
            usage.input_tokens_details, "cached_tokens", 0
        )
        rate = cached_tokens / total_input if total_input > 0 else 0

        entry = {
            "total_input": total_input,
            "cached_tokens": cached_tokens,
            "cache_rate": rate,
        }
        self.calls.append(entry)
        return entry

    @property
    def average_rate(self) -> float:
        if not self.calls:
            return 0.0
        return sum(a["cache_rate"] for a in self.calls) / len(self.calls)

    def report(self) -> str:
        if not self.calls:
            return "No calls."
        return (
            f"Calls: {len(self.calls)} | "
            f"Average cache rate: {self.average_rate:.1%} | "
            f"Tokens saved: "
            f"{sum(a['cached_tokens'] for a in self.calls):,}"
        )

monitor = CacheMonitor()

Three patterns that make the difference

The first pattern concerns processing order. When you process a batch of heterogeneous requests, running them in arrival order alternates prefixes and prevents the cache from staying warm. Grouping by shared context solves the problem: gather first, execute later, so each group benefits from the prefix primed by its first request.

from collections import defaultdict

def group_by_context(
    requests: list[dict],
) -> dict[str, list[dict]]:
    """Groups requests by common prefix."""
    groups = defaultdict(list)

    for request in requests:
        # Key = hash of shared context
        key = hash(request.get("shared_context", ""))
        groups[key].append(request)

    return dict(groups)

def process_by_cohort(requests: list[dict]):
    """Processes grouped requests to maximize caching."""
    groups = group_by_context(requests)

    for key, group in groups.items():
        # All requests in the group share the same prefix
        for request in group:
            response = client.responses.create(
                model="gpt-5.6-terra",
                instructions=request["shared_context"],
                input=request["question"],
            )
            monitor.record(response.usage)

The second pattern handles cold starts. The first request in a series pays full price since nothing is cached yet; if that first request comes from a real user on a Monday morning at 9 a.m., they bear the latency. A warm-up request sent before the service opens shifts that cost to where it bothers no one. Note the guard at the start of the function: below 1,024 tokens, caching doesn’t apply, and pre-heating would be pure expense.

def warm_up_cache(system_prompt: str):
    """Sends a minimal request to prime the cache."""
    if len(system_prompt) // 4 < 1024:
        print("Prompt too short for caching (< 1,024 tokens)")
        return

    response = client.responses.create(
        model="gpt-5.6-terra",
        instructions=system_prompt,
        input="Confirm that you are ready.",
        max_output_tokens=10,
    )

    usage = response.usage
    print(
        f"Cache primed: {usage.input_tokens} input tokens, "
        f"{getattr(usage.input_tokens_details, 'cached_tokens', 0)} cached"
    )

The third pattern makes cache invalidation visible. Fixing a comma in a 5,000-token system prompt seems trivial, yet that comma flushes the cache for everyone. By attaching a fingerprint to the content, you detect the change at deployment rather than the next day on the bill, and you can immediately follow up with pre-heating.

import hashlib

class VersionedPrompt:
    """Manages prompt versions to control caching."""

    def __init__(self, content: str):
        self.content = content
        self.version = hashlib.md5(content.encode()).hexdigest()[:8]

    def is_same_version(self, other: "VersionedPrompt") -> bool:
        return self.version == other.version

# If the version changes, the cache will be invalidated
prompt_v1 = VersionedPrompt("Instructions v1...")
prompt_v2 = VersionedPrompt("Instructions v2 modified...")

if not prompt_v1.is_same_version(prompt_v2):
    print("Warning: prompt changed, cache will be invalidated")
    warm_up_cache(prompt_v2.content)

What this means in volume

Take a real-world case of 10,000 calls per day with a 3,000-token system prompt. Without caching, you bill 10,000 × 3,000, or 30M input tokens every day. With an 80% hit rate, 6M tokens remain at full price and 24M drop to half price, bringing the savings to about 40% on input tokens. The figure seems modest relative to a single call; over a year of production, it changes the application’s cost structure.

Key takeaways

• Structure your prompts in layers from most stable to most variable
• Group requests by shared context to keep the cache warm
• Warm up the cache before traffic spikes
• Monitor the cache rate with input_tokens_details.cached_tokens
• Version your prompts to anticipate cache invalidations