Token counting and budgeting

Understanding tokens

Tokens are the fundamental unit of measurement for LLMs. Every API call is billed in tokens, and every model rejects requests that exceed its context window. Mastering token counting lets you do three things that approximation cannot: predict costs before you incur them, optimize your prompts based on hard numbers rather than gut feel, and respect context limits without discovering an overflow only when the API throws an error.

Counting with tiktoken

The tiktoken library applies the exact same tokenization as the model. You instantiate an encoder for the target model, then count the length of the encoded sequence.

import tiktoken

# Encoder for GPT models
encoder = tiktoken.encoding_for_model("gpt-5.6-terra")

def count_tokens(text: str) -> int:
    """Counts the number of tokens in a text."""
    return len(encoder.encode(text))

# Examples
print(count_tokens("Hello"))           # ~1-2 tokens
print(count_tokens("Hello world"))     # ~3-4 tokens
print(count_tokens("Development"))     # ~2-3 tokens (accents cost more)

The third example deserves a pause. “Development” costs more than its character length would suggest because accented characters tokenize less efficiently. In practice, a French corpus consumes more tokens than an equivalent English one: if you estimated your budget using English prompts and then deployed in French, your actual bill will exceed your forecast.

An API call carries more than raw text: each message includes a role, sometimes a name, and the API adds structural tokens. The following function incorporates this overhead to provide a realistic total.

def count_conversation_tokens(messages: list[dict]) -> int:
    """Counts tokens in a conversation (messages + overhead)."""
    total = 0
    for message in messages:
        total += 4  # Overhead per message (role, etc.)
        total += count_tokens(message["content"])
        if message.get("name"):
            total += count_tokens(message["name"])
    total += 2  # End tokens
    return total

conversation = [
    {"role": "system", "content": "You are a technical assistant."},
    {"role": "user", "content": "Explain microservices."},
    {"role": "assistant", "content": "Microservices are an architecture..."},
]

print(f"Total tokens: {count_conversation_tokens(conversation)}")

On a three-message conversation, the overhead is negligible. On a history of two hundred short messages—typical for a support chat—it adds up to nearly a thousand tokens you didn’t account for.

Budgeting before calling

The context window isn’t an input budget: it’s a shared budget between what you send and what the model generates. If you fill the window to the brim with your prompt, the response will be truncated. The BudgetTokens structure makes this allocation explicit by reserving a portion for the output upfront.

from dataclasses import dataclass

@dataclass
class BudgetTokens:
    """Manages the token budget for a call."""
    context_limit: int       # Model window
    system_tokens: int = 0   # System prompt
    history_tokens: int = 0  # Conversation history
    user_tokens: int = 0     # Current message
    output_reserve: int = 2000 # Reserved for response

    @property
    def available_tokens(self) -> int:
        used = (
            self.system_tokens
            + self.history_tokens
            + self.user_tokens
        )
        return self.context_limit - used - self.output_reserve

    @property
    def is_over(self) -> bool:
        return self.available_tokens < 0

    def report(self) -> str:
        return (
            f"Context: {self.context_limit} | "
            f"System: {self.system_tokens} | "
            f"History: {self.history_tokens} | "
            f"User: {self.user_tokens} | "
            f"Output reserve: {self.output_reserve} | "
            f"Available: {self.available_tokens}"
        )

# Example with a 128K token context window
budget = BudgetTokens(
    context_limit=128_000,
    system_tokens=count_tokens(system_prompt),
    history_tokens=count_conversation_tokens(history),
    user_tokens=count_tokens(user_message),
)

if budget.is_over:
    print("Budget exceeded — compaction required")

The report method does more than aid debugging. Log it on a sample of production requests, and you’ll immediately see which component is consuming your context. In most applications, the surprise comes from the history, rarely from the system prompt that everyone monitors.

Next, you need to act when the budget is exceeded. The prepare_context function applies the simplest and most robust rule: keep the most recent messages and drop the oldest ones as soon as the budget is exhausted. The iteration runs in reverse, inserting at the head to restore chronological order.

def prepare_context(
    system_prompt: str,
    history: list[dict],
    message: str,
    model_limit: int = 128_000,
    output_reserve: int = 4_000,
) -> list[dict]:
    """Prepares the context while respecting the token budget."""

    system_tokens = count_tokens(system_prompt)
    message_tokens = count_tokens(message)
    history_budget = (
        model_limit - system_tokens - message_tokens - output_reserve
    )

    # Keep the most recent messages within budget
    kept_messages = []
    used_tokens = 0

    for msg in reversed(history):
        msg_tokens = count_tokens(msg["content"]) + 4
        if used_tokens + msg_tokens > history_budget:
            break
        kept_messages.insert(0, msg)
        used_tokens += msg_tokens

    return (
        [{"role": "system", "content": system_prompt}]
        + kept_messages
        + [{"role": "user", "content": message}]
    )

This truncation is brutal: anything that falls outside the budget is lost. When older context truly matters—a client file tracked over several weeks—replace pure deletion with the compaction strategy covered in the previous lesson.

Estimating cost before spending

The most expensive billing errors come from batch processing: you run a pass on 10,000 documents and discover the amount the next day. Estimating first, in a few milliseconds of local computation, avoids this scenario.

def estimate_cost(
    input_tokens: int,
    estimated_output_tokens: int,
    model: str = "gpt-5.6-terra",
) -> dict:
    """Estimates the cost of a call before executing it."""
    # Rates per million tokens (July 2026 — check official pricing)
    rates = {
        "gpt-5.6-luna": {"input": 0.20, "output": 1.20},
        "gpt-5.6-terra": {"input": 2.00, "output": 12.00},
        "gpt-5.6-sol": {"input": 5.00, "output": 30.00},
    }

    rate = rates.get(model, rates["gpt-5.6-terra"])
    input_cost = (input_tokens / 1_000_000) * rate["input"]
    output_cost = (estimated_output_tokens / 1_000_000) * rate["output"]

    return {
        "model": model,
        "input_tokens": input_tokens,
        "estimated_output_tokens": estimated_output_tokens,
        "input_cost": f"${input_cost:.6f}",
        "output_cost": f"${output_cost:.6f}",
        "total_cost": f"${input_cost + output_cost:.6f}",
    }

# Before sending a large prompt
estimation = estimate_cost(
    input_tokens=50_000,
    estimated_output_tokens=2_000,
    model="gpt-5.6-sol",
)
print(estimation)

The rate table is hardcoded in this example, with the date in a comment. Treat it as configuration data to verify regularly against the official pricing grid: an estimate calculated on outdated prices is more dangerous than no estimate at all, because it inspires false confidence.

Finally, know the limits of each model to plan your budget. Context windows vary from model to model within the GPT-5.6 family—consult the official OpenAI models page for current values, and configure context_limit from your settings rather than hardcoding it in the calling code.

Key takeaways

• Use tiktoken to count tokens before every call, accounting for per-message overhead
• French consumes more tokens than English for equivalent content: adjust your estimates
• Always reserve a budget for output tokens within the context window
• Manage context automatically by dropping the oldest messages
• Estimate costs before sending expensive requests, using verified rates