Optimize Cache Hit Rate

Structure Your Prompts for Caching

The Grok API cache works via prefix matching. Tokens are cached in order, from first to last, and matching stops at the first differing character. The consequence is severe: modify a token in the middle of your prompt, and everything after it loses its cache, even if the rest is word-for-word identical. A twenty-character timestamp at the start of a prompt is enough to trigger full pricing on the 50,000 tokens that follow. Structuring your prompts isn’t engineering vanity—it’s what separates a $200 bill from a $1,000 bill for the same service.

The Golden Rule: Do Not Modify History

The most common pitfall in production is tweaking the message history between requests. Reformatting a previous message, reordering turns, injecting a current date or session ID into the system prompt: each of these gestures, seemingly harmless, invalidates the cache from the modification point onward. The most widespread case fits in a single line of code:

# BAD: timestamp in system prompt
messages = [
    {"role": "system", "content": f"Date: {datetime.now()}. You are an assistant."},
    # ...
]

This timestamp changes on every call. Since it sits in the very first message, prefix matching fails on the first token, and the entire prompt is billed at the standard rate, conversation after conversation. If your model genuinely needs to know the date, place it in the last user message, where it costs only a few dozen uncached tokens.

The correct version of the same prompt keeps a strictly fixed header, an append-only history, and concentrates all novelty at the end:

# GOOD: static content first, variables at the end
messages = [
    {"role": "system", "content": "You are a customer support specialist assistant."},
    # UNCHANGED conversation history
    {"role": "user", "content": "Previous question..."},
    {"role": "assistant", "content": "Previous response..."},
    # Only the last message is new
    {"role": "user", "content": "User's new question"}
]

A Prompt Architecture That Caches Well

The principle fits in one sentence: from most stable to most variable. Concretely, four layers follow in this order.

1. System prompt (stable): general instructions, persona, rules
2. Reference context (stable): documentation, knowledge base
3. Conversation history (incremental): append-only, never modify
4. Current message (variable): the user’s new request

With this arrangement, layers 1 to 3 fall into the cache, and only layer 4 is billed at the standard rate. On an application where the system prompt and repository weigh 8,000 tokens, a 60-token user question bills only those 60 tokens at full price.

This logic extends to less obvious cases. If your application relies on a long reference context—product documentation, FAQ, technical specifications—place it right after the system prompt: it will be cached from the second request of each conversation onward. And contrary to common assumption, the benefit isn’t confined to a single session: two distinct conversations that share the same prefix, same system prompt, and same reference context, cache each other. In a multi-user application with a common prompt, each user benefits from the others’ traffic, making a long, stable system prompt far less costly than it appears.

The Reflex That Breaks Everything: Truncating History

When a conversation lengthens and approaches the context limit, the instinct is to cut the earliest messages. This is exactly what you must not do: by removing the beginning, you shift everything else and destroy prefix matching. You save a few thousand tokens and pay back fifty thousand at full price. Prefer summarization, which condenses old exchanges into a rewritten system prompt once, leaving recent messages intact:

# Instead of truncating history:
# messages = messages[-10:]  # BAD: breaks cache

# Summarize old messages into the system prompt:
messages = [
    {"role": "system", "content": "Instructions... Summary of previous exchanges: ..."},
    # Recent messages unmodified
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."},
    {"role": "user", "content": "New question"}
]

Measure and Improve

Track the daily cache hit rate as a production metric in its own right, just like an error rate. Here is how to interpret the value you observe.

The value of this tracking is mostly differential. A stable value at 65% that drops to 20% overnight points to the faulty deployment within hours: in almost all cases, a variable was just introduced at the head of the prompt. Test it yourself on your current application—take the average rate over one hundred requests, move a variable element to the end of the prompt, and compare.

Key Takeaways

• Cache works via prefix matching: never modify tokens at the start of the prompt
• Structure your prompts from most stable to most variable
• Never add variable data (timestamp, IDs) in the system prompt
• Prefer summarization over truncation to preserve cache on long conversations
• Monitor your daily cache hit rate as a financial performance metric