Temperature and Top P in Depth
On this page
Mastering Output Randomization
You covered the basics of temperature and top_p in Lesson 3. Now it’s time to go deeper: how these two parameters mathematically affect generation, how to calibrate them for specific use cases, and why combining them is a classic source of production issues.
How Temperature Works
At each generation step, the model calculates a probability distribution over the entire vocabulary: which token comes next, and with what likelihood. Temperature intervenes on this distribution before sampling, altering its shape.
Between 0.0 and 0.3, the distribution becomes “spiky”: the most probable token massively dominates the others, making the model quasi-deterministic. This is what you want for extraction or classification, where two identical calls must produce identical results. Between 0.4 and 0.8, a balance emerges between coherence and variety—this is the most common range in production, used for chatbots and assisted writing. Beyond that, from 0.9 to 1.5, the distribution flattens: significantly less probable tokens now have a realistic chance of being selected. Creativity increases, but so does the risk of incoherence, and both rise together.
The following experiment makes this phenomenon tangible using a deliberately mundane sentence.
from mistralai.client import Mistral
import os
client = Mistral(api_key=os.getenv("MISTRAL_API_KEY"))
prompt = [
{"role": "system", "content": "Complete the sentence with a single word."},
{"role": "user", "content": "The color of the sky is..."}
]
# Temperature 0 — always the same response
for i in range(3):
r = client.chat.complete(model="mistral-large-latest", messages=prompt, temperature=0.0, max_tokens=5)
print(f"T=0.0 trial {i+1}: {r.choices[0].message.content}")
# blue, blue, blue
# Temperature 1.5 — varied responses
for i in range(3):
r = client.chat.complete(model="mistral-large-latest", messages=prompt, temperature=1.5, max_tokens=5)
print(f"T=1.5 trial {i+1}: {r.choices[0].message.content}")
# blue, azure, turquoise (varies)One clarification is necessary regarding temperature=0: even at this value, slight variations can appear between two calls. The cause lies not in the model but in the hardware—floating-point operations on GPUs do not guarantee an identical evaluation order on every execution, and a rounding discrepancy is sometimes enough to tip the choice between two nearly equiprobable tokens. If your use case demands maximum reproducibility, combine temperature=0 with a random_seed when the model supports it.
How Top P Works (Nucleus Sampling)
Top P acts after temperature, on the distribution it has already reshaped. Its principle is not to distort probabilities but to truncate the list of candidates, keeping only the tokens whose cumulative probability sum reaches the threshold P. The full sequence unfolds in four steps:
- The model calculates the probabilities for each token
- Temperature modifies these probabilities
- Tokens are sorted by decreasing probability
- Top P retains tokens until the cumulative threshold is reached
Concretely, a top_p of 0.1 leaves only a handful of candidates alive—useful when the expected answer is factual and nearly unique. At 0.95, almost the entire plausible vocabulary remains in contention, which suits a request for invention.
# Top P = 0.1 — very restrictive, only the most probable tokens
response = client.chat.complete(
model="mistral-large-latest",
messages=[{"role": "user", "content": "The capital of France is"}],
temperature=0.7,
top_p=0.1 # Only the top 10% of the probability mass
)
# Top P = 0.95 — broad, almost all tokens considered
response = client.chat.complete(
model="mistral-large-latest",
messages=[{"role": "user", "content": "Invent a new word for a feeling."}],
temperature=0.7,
top_p=0.95
)The Golden Rule: One Parameter at a Time
Mistral explicitly recommends not adjusting temperature and top_p simultaneously. The reason follows from what precedes: both act on the same distribution, one by distorting it, the other by cutting it. Their effects multiply in unpredictable ways, and you lose any ability to attribute a behavioral change to one or the other during testing. Choose your lever, and leave the other at its default value.
# GOOD — adjust temperature, top_p default
response = client.chat.complete(
model="mistral-large-latest",
messages=messages,
temperature=0.3 # top_p remains at 1.0 by default
)
# GOOD — adjust top_p, temperature default
response = client.chat.complete(
model="mistral-large-latest",
messages=messages,
top_p=0.5 # temperature remains at 0.7 by default
)
# AVOID — both at the same time
response = client.chat.complete(
model="mistral-large-latest",
messages=messages,
temperature=0.2,
top_p=0.5 # Unpredictable interaction
)Calibrating by Use Case
Rather than choosing a temperature ad hoc, name your configurations and reason by objective. Extraction and classification aim for an exact, reproducible response, so 0.0. Summarization stays faithful to the source text with a bit of fluidity, so 0.3. Assisted writing seeks naturalness without superfluous creativity, at 0.5; conversational chatbots need varied responses, at 0.7; brainstorming demands maximum creativity, at 1.2. In all cases, top_p remains at 1.0, in accordance with the previous rule.
# Data Extraction / Classification
# Objective: exact, reproducible response
CONFIG_EXTRACTION = {"temperature": 0.0, "top_p": 1.0}
# Summarization / Reformulation
# Objective: faithful to source text with some fluidity
CONFIG_SUMMARY = {"temperature": 0.3, "top_p": 1.0}
# Assisted Writing / Professional Email
# Objective: natural but not too creative
CONFIG_WRITING = {"temperature": 0.5, "top_p": 1.0}
# Conversational Chatbot
# Objective: varied and natural responses
CONFIG_CHATBOT = {"temperature": 0.7, "top_p": 1.0}
# Brainstorming / Idea Generation
# Objective: maximum creativity
CONFIG_CREATIVE = {"temperature": 1.2, "top_p": 1.0}
# Usage
def smart_complete(client, messages, task_type="chatbot"):
configs = {
"extraction": CONFIG_EXTRACTION,
"summary": CONFIG_SUMMARY,
"writing": CONFIG_WRITING,
"chatbot": CONFIG_CHATBOT,
"creative": CONFIG_CREATIVE,
}
config = configs.get(task_type, CONFIG_CHATBOT)
return client.chat.complete(
model="mistral-large-latest",
messages=messages,
**config
)Experiment Methodically
These values are starting points, not absolute truths. To find the right setting for your own task, you must measure, and a single trial per temperature measures nothing. The function below runs five generations for each tested value and counts the unique responses: this is your diversity indicator. A task that yields 5/5 unique responses at T=0.3 signals a prompt that is too open, far more than a poorly chosen temperature.
def test_temperature_range(client, messages, temperatures):
"""Tests multiple temperatures on the same prompt."""
results = []
for temp in temperatures:
responses = []
for _ in range(5): # 5 trials per temperature
r = client.chat.complete(
model="mistral-large-latest",
messages=messages,
temperature=temp,
max_tokens=200
)
responses.append(r.choices[0].message.content)
# Measure diversity
unique = len(set(responses))
results.append({
"temperature": temp,
"unique_responses": unique,
"sample": responses[0][:100]
})
for r in results:
print(f"T={r['temperature']:.1f} | {r['unique_responses']}/5 unique | {r['sample']}...")
# Test
test_temperature_range(
client,
[{"role": "user", "content": "Give a productivity tip."}],
[0.0, 0.3, 0.7, 1.0, 1.5]
)Key Takeaways
- Temperature modifies the shape of the token probability distribution
- Top P truncates the distribution by keeping the most probable tokens
- Never adjust both simultaneously—modify one parameter, keep the other default
temperature=0is not perfectly deterministic due to GPU rounding- Calibrate with methodical testing: 5+ trials per configuration, measure diversity