Optimize the cache

Cache: A key performance factor

Cache hits are the primary driver of inference speed for grok-build-0.1. When the start of your request (the prefix) matches a previous request, the model retrieves already computed calculations from the cache. Result: the response arrives much faster.

Understanding this mechanism lets you optimize your interactions for near-instant responses.

How the cache works

The cache operates on the common prefix principle:

1. You send a prompt with conversation history
2. The model computes tokens from the start (system prompt + history)
3. If this prefix was computed recently, it is retrieved from the cache
4. Only new tokens (your last message) require full computation

The longer the common prefix, the greater the gain. In a pair programming scenario where you exchange several messages, each request reuses all previous history.

The golden rule: Do not modify the history

This is the most important point in this lesson. Avoid modifying the prompt history between two requests. Any modification to the start of the conversation invalidates the cache.

What breaks the cache

• Reordering messages in history
• Modifying a previous message (even a single character)
• Deleting messages from history
• Inserting a message in the middle of the conversation
• Changing the system prompt during the conversation

What preserves the cache

• Adding a new message to the end of the conversation (normal behavior)
• Keeping the system prompt identical throughout the session
• Keeping the history intact without modification

The stable prefix

To maximize cache hits, structure your requests with a stable prefix:

[System prompt (fixed)]
[User message 1 (fixed)]
[Model response 1 (fixed)]
[User message 2 (fixed)]
[Model response 2 (fixed)]
...
[New user message] ← only part computed

The system prompt is particularly important. Define it once at the start of the session and do not change it. If you are developing an API agent with grok-build-0.1, write a complete and stable system prompt.

Real-world impact

In a typical pair programming scenario with 10 exchanges:

• Without cache optimization: each request recomputes the entire history. Latency increases with each exchange
• With optimized cache: only the last message is computed. Latency remains constant, regardless of conversation length

The difference can be on the order of a few seconds per interaction. Over a one-hour development session with hundreds of exchanges, the cumulative gain is considerable.

For agent developers

If you are building an API agent that uses grok-build-0.1 in sequential tool usage scenarios:

• The prefix remains identical between each tool call, which is retrieved from cache automatically
• Do not reformat tool results before sending them back to the model
• Avoid history transformations (summarization, compression) that would break the cache
• Keep the same message order on each call

Key takeaways

• Cache hits are the primary speed factor for grok-build-0.1
• Never modify the prompt history between two requests
• The system prompt must remain stable throughout the session
• Add to the end, never modify the start
• In sequential agents, the cache works naturally if history is preserved