Tokens, Context, and Conversation Window

The token is the unit the model manipulates and the unit your bill tracks. As long as this concept remains fuzzy, your costs will too, and you will eventually hit a context limit error without knowing why. This lesson clears up both issues.

What Is a Token?

A token is not a word. It is a fragment of text that the model treats as an indivisible unit. In French, one token represents about 0.7 words on average. The tiktoken library lets you see exactly how text splits: notice how “allez-vous” breaks into two tokens while “aujourd’hui,” which appears more often in the training corpus, fits into one. This detail explains why a French text consumes more tokens than an English text of comparable length.

from openai import OpenAI
client = OpenAI()

# To estimate tokens, use the tiktoken library
import tiktoken

# cl200k_base is used here as an example: each model generation has its own encoder.
# Check the encoder for the model you call before relying on the count.
enc = tiktoken.get_encoding("cl200k_base")

texte = "Bonjour, comment allez-vous aujourd'hui ?"
tokens = enc.encode(texte)
print(f"Text: {texte}")
print(f"Number of tokens: {len(tokens)}")
print(f"Tokens: {tokens}")
print(f"Decoded: {[enc.decode([t]) for t in tokens]}")

# Result:
# Text: Bonjour, comment allez-vous aujourd'hui ?
# Number of tokens: 9
# Tokens: [38025, 11, 4068, 67590, 5072, 76439, 30, ...]
# Decoded: ['Bonjour', ',', ' comment', ' allez', '-vous', " aujourd'hui", ' ?']

Always verify which encoder matches the model you call. A count performed with the wrong encoder remains a useful approximation but will not match your bill exactly.

The Context Window

Every model has a maximum context window, and the trap for beginners is that it includes both input and output. This is not an input budget: it is a total budget. If your prompt takes up 190,000 tokens of a 200,000-token window, only 10,000 tokens remain for the response. The model will cut off mid-sentence with no further warning.

# Previous generation, for reference:
# GPT-5.4: 1,000,000 context tokens
# GPT-5.3: 200,000 context tokens
# gpt-5.6-sol  : 200,000 context tokens
# gpt-5.6-terra : 200,000 context tokens

# Example: if your input is 190,000 tokens in a 200,000-token window,
# only 10,000 tokens remain for the response

response = client.responses.create(
    model="gpt-5.6-terra",
    input="A very long text...",
    max_output_tokens=10000  # Limit explicitly
)

After the fact, you can read the exact count in response.usage, which distinguishes input tokens, output tokens, and the total:

response = client.responses.create(
    model="gpt-5.6-terra",
    input="Explain tokens in three sentences."
)

usage = response.usage
print(f"Input tokens: {usage.input_tokens}")
print(f"Output tokens: {usage.output_tokens}")
print(f"Total: {usage.total_tokens}")

# Result:
# Input tokens: 12
# Output tokens: 58
# Total: 70

Three Optimization Levers

The first lever is prompt conciseness. Polite formulas and rhetorical precautions do not improve the response and cost tokens down to the last one: the first version below uses thirty-five tokens for a result identical to the second, which uses ten. At the scale of a service handling a million requests, the difference is no longer anecdotal.

# BAD — verbose prompt (35 tokens)
response = client.responses.create(
    model="gpt-5.6-terra",
    input="Could you please give me a list "
          "of three fruits that are red in color, if that "
          "does not bother you of course?"
)

# GOOD — concise prompt (10 tokens)
response = client.responses.create(
    model="gpt-5.6-terra",
    input="List 3 red fruits."
)

The second lever targets output, which costs several times more than input. Combine the instruction in the prompt with a technical ceiling: the first frames the writing, the second guarantees your budget holds even if the model ignores the instruction.

# No limit — the model can generate many tokens
response = client.responses.create(
    model="gpt-5.6-terra",
    input="Explain photosynthesis."
)

# With limit — cost control
response = client.responses.create(
    model="gpt-5.6-terra",
    input="Explain photosynthesis in 2 sentences maximum.",
    max_output_tokens=100
)

The third lever is estimating cost before the call, which becomes essential as soon as you process documents of variable size. You can then reject or split a request that is too heavy rather than discovering it on your bill.

import tiktoken

def estimate_cost(text: str, model: str = "gpt-5.6-terra") -> dict:
    """Estimate the cost of an API call."""
    enc = tiktoken.get_encoding("cl200k_base")
    input_tokens = len(enc.encode(text))

    # Approximate prices per million tokens (check current prices)
    prices = {
        "gpt-5.6-sol": {"input": 5.0, "output": 30.0},
        "gpt-5.6-terra": {"input": 2.0, "output": 12.0},
        "gpt-5.6-luna": {"input": 0.20, "output": 1.20},
    }

    input_cost = (input_tokens / 1_000_000) * prices[model]["input"]

    return {
        "input_tokens": input_tokens,
        "input_cost_usd": round(input_cost, 6),
        "model": model
    }

print(estimate_cost("Explain relativity in 3 sentences."))
# Result: {'input_tokens': 10, 'input_cost_usd': 2.5e-05, 'model': 'gpt-5.6-terra'}

Context Accumulates Over the Conversation

When you chain exchanges using previous_response_id, each turn restarts from the entirety of previous turns. Input tokens therefore keep growing: a dialogue that starts at 50 tokens weighs 150 on the second turn and much more on the tenth. This is the most common cause of bills spiraling without increased traffic, and monitoring response.usage.input_tokens is your warning signal.

# Turn 1: 50 tokens
r1 = client.responses.create(
    model="gpt-5.6-terra",
    input="Hello, I am a Python developer."
)

# Turn 2: 50 + 100 = 150 tokens (cumulative context)
r2 = client.responses.create(
    model="gpt-5.6-terra",
    input="Which web frameworks do you recommend?",
    previous_response_id=r1.id
)

# Turn 10: tokens accumulate!
# Monitor response.usage.input_tokens to avoid surprises
print(f"Cumulative tokens at turn 2: {r2.usage.input_tokens}")

For long conversations, the workaround is to actively truncate history: remove the oldest messages until you drop below a chosen limit, while keeping the first one, which usually carries the founding context of the exchange.

def conversation_with_limit(messages: list, token_limit: int = 50000):
    """Keep the conversation under a token limit."""
    enc = tiktoken.get_encoding("cl200k_base")

    total = sum(len(enc.encode(m["content"])) for m in messages)

    # Remove oldest messages if necessary
    while total > token_limit and len(messages) > 2:
        removed = messages.pop(1)  # Keep the first message (context)
        total -= len(enc.encode(removed["content"]))

    return messages

Key Takeaways

One token equals about 0.7 words in French
The context window includes input AND output
Windows vary by model: in the previous generation, 1M tokens for GPT-5.4 and 200K for GPT-5.3, o3-pro, and o4-mini
Use tiktoken to count tokens before the call
Monitor response.usage to control your costs
Actively manage context in long conversations