Tokenization: Understanding How the Model Sees Your Prompt

Tokenization: Understanding How the Model Sees Your Prompt

The model doesn’t read text; it processes tokens. This distinction isn’t academic. It determines what you pay, how much fits in your context window, and sometimes why the model stumbles over a word you thought was trivial. Understanding tokenization gives you three concrete levers for your application: optimizing your prompts, controlling your costs, and explaining behaviors that seem inexplicable when you think in words.

What Is a Token?

A token is the basic unit the model manipulates. It’s neither a word nor a character, but a text fragment produced by an encoding algorithm called BPE (Byte Pair Encoding), which iteratively merges the most frequent character sequences in the training corpus. On average, one token represents about 4 characters in English, but this ratio varies significantly by language. The script below demonstrates this with an English sentence and its French equivalent, showing the actual split. Seeing the tokenizer cut “aujourd’hui” (today) is worth more than any theoretical explanation.

import tiktoken

# o200k encoding, used by recent GPT models
enc = tiktoken.get_encoding("o200k_base")

texte_en = "Hello, how are you today?"
texte_fr = "Bonjour, comment allez-vous aujourd'hui ?"

tokens_en = enc.encode(texte_en)
tokens_fr = enc.encode(texte_fr)

print(f"English: {len(tokens_en)} tokens for {len(texte_en)} characters")
print(f"French: {len(tokens_fr)} tokens for {len(texte_fr)} characters")

# Visualize the split
for t in tokens_fr:
    print(f"  Token {t}: '{enc.decode([t])}'")

The Impact of Language on Cost

French consumes 20% to 30% more tokens on average than English to express the same idea. This happens because French is less represented in the tokenizer’s training corpus, so its words are split more often. The first consequence is financial: you pay per input and output token, so a French-language knowledge base costs mechanically more than its English translation for identical content. The second consequence affects the context window, which fills up faster with French documents—a conversation history that held thirty exchanges in English might only hold twenty-four in French. The third is latency: since each output token is generated sequentially, a French response takes longer to display than an English response of equivalent content.

Three Costly Pitfalls

Rare words and neologisms top the list. Because they appear infrequently in the training corpus, they get split into sub-tokens, which can degrade the model’s comprehension. This applies especially to your internal product references and homegrown code names that the tokenizer has never seen:

# "Anticonstitutionnellement" (Unconstitutionally) will be split into multiple tokens
tokens = enc.encode("Anticonstitutionnellement")
print(f"{len(tokens)} tokens")
for t in tokens:
    print(f"  '{enc.decode([t])}'")

JSON and structured code hold a similar surprise. Braces, brackets, and quotes each consume a token, meaning indented JSON costs significantly more than its minified version. In a few-shot prompt carrying several payload examples, you don’t pay this difference once; you pay it on every call, all day long.

json_verbose = '''{
    "nom": "Dupont",
    "prenom": "Jean",
    "age": 42
}'''

json_compact = '{"nom":"Dupont","prenom":"Jean","age":42}'

print(f"Verbose: {len(enc.encode(json_verbose))} tokens")
print(f"Compact: {len(enc.encode(json_compact))} tokens")

The most subtle pitfall is formatting itself: leading spaces, blank lines, and indentation all count. A system prompt carefully aligned for team readability costs more, for strictly identical content, than a compact prompt. This isn’t a reason to write illegible code, but it is a reason to make that trade-off knowingly.

Optimizing Your Prompts

The golden rule is to measure before you optimize: tiktoken gives you the exact count, while intuition is often off by a factor of two. Once you’ve measured, prefer concise instructions. “Respond in JSON” says exactly what “I would like you to respond using the JSON format” says, for one-third the tokens. In context blocks, abbreviations work well: the model interprets “Q:” and “A:” just as reliably as “Question:” and “Answer:”. Finally, compress business context by summarizing long documents upstream, rather than copying ten pages of documentation into every request when you only need one paragraph.

Cost Calculation

def estimate_cost(prompt: str, estimated_response: int = 500,
                 model: str = "gpt-5.6-terra") -> dict:
    """Estimates the cost of an API call."""
    enc = tiktoken.get_encoding("o200k_base")
    tokens_in = len(enc.encode(prompt))

    # GPT-5.6 Terra pricing, July 2026 (verify current rates)
    price_input = 2.50 / 1_000_000   # per token
    price_output = 15.00 / 1_000_000  # per token

    cost = (tokens_in * price_input) + (estimated_response * price_output)

    return {
        "tokens_input": tokens_in,
        "tokens_output_estimated": estimated_response,
        "estimated_cost_usd": round(cost, 6)
    }

This function quickly becomes a project utility: hook it up to your prompts before deployment, and it tells you what a day of traffic will cost while there’s still time to act. Extend this with an exercise that proves more instructive than it looks. Install tiktoken (pip install tiktoken), tokenize your most complex system prompt, and identify the most expensive segments—almost always the few-shot examples and JSON blocks. Then rewrite it aiming for 20% fewer tokens without losing a single piece of information. You’ll discover along the way how many sentences in your prompt were just there to reassure you.

Rates as of August 5, 2026 — prices change regularly. Before budgeting, verify the current rate card on the official OpenAI pricing page.

Key Takeaways

• A token is not a word; it’s a fragment from BPE
• French costs 20–30% more tokens than English
• Systematically measure your prompts with tiktoken
• Optimize JSON and formatting to reduce costs
• The context window is measured in tokens, not words