Top P and Nucleus Sampling

A Second Lever to Control Generation

The previous lesson gave you the first lever for controlling generation: temperature. Top P, also called nucleus sampling, is its natural complement. While temperature changes the entire probability distribution, Top P acts as a filter that limits the number of candidate tokens. Both parameters influence response diversity, but through very different mechanisms—and it’s precisely this difference that will let you fine-tune Mistral model behavior instead of twiddling knobs at random.

How Top P Works

Let’s walk through the generation process again. At each step, the model calculates a probability for every possible token. Top P steps in during three stages: first, probabilities are adjusted by temperature; then, tokens are sorted by descending probability and their probabilities are summed cumulatively; finally, only tokens whose cumulative probability does not exceed the Top P threshold are kept. The final draw happens only among these survivors.

An example makes the mechanism clear. Imagine the model must choose the next word after “The cat is on the”:

With Top P = 0.75, only “roof”, “bed”, and “sofa” are kept, since their cumulative sum hits exactly 0.75: the model will choose among these three options only. With Top P = 0.90, “floor” and “ledge” come into play as well. The general rule is clear from the table: the higher the Top P, the more candidates are considered.

Top P vs Temperature: What’s the Difference?

Since both parameters control diversity, you might think they’re interchangeable. They aren’t. Temperature changes the shape of the distribution: a high value flattens the curve and makes improbable tokens gradually more accessible, without ever excluding any. Top P, on the other hand, cuts the distribution: tokens below the threshold are ignored outright, regardless of their residual probability.

Keep this image in mind: temperature is a “soft” setting, which adjusts gradually, while Top P is a “sharp” setting, which includes or excludes. In creative text, a high temperature with a moderate Top P allows for fantasy while preventing drift toward truly improbable tokens—this is often the most comfortable combination.

Using Top P with the Mistral API

In code, the parameter is simply added to the call, alongside temperature:

from mistralai.client import Mistral

client = Mistral(api_key="your-api-key")

response = client.chat.complete(
    model="mistral-large-latest",
    messages=[
        {"role": "user", "content": "Propose a name for an AI startup."}
    ],
    temperature=0.7,
    top_p=0.9
)

When to Adjust Top P

A low Top P, between 0.1 and 0.5, concentrates the model on the most probable options—maximum precision. This is the right choice when the expected answer belongs to a closed set, like in this strict classification:

response = client.chat.complete(
    model="mistral-large-latest",
    messages=[
        {"role": "system", "content": "Respond only with: positive, neutral, or negative."},
        {"role": "user", "content": "Classify: The product arrived in good condition."}
    ],
    temperature=0.0,
    top_p=0.3
)

A medium Top P, between 0.5 and 0.8, suits most professional tasks: the text stays coherent without becoming monotone. This is the regime for a report, a summary, or a document introduction:

response = client.chat.complete(
    model="mistral-large-latest",
    messages=[
        {"role": "user", "content": "Write the introduction for a report on cybersecurity in SMEs."}
    ],
    temperature=0.5,
    top_p=0.7
)

Finally, a high Top P, between 0.8 and 1.0, opens the field for creativity and idea exploration—you accept less probable candidates to get proposals that are outside the box:

response = client.chat.complete(
    model="mistral-large-latest",
    messages=[
        {"role": "user", "content": "Invent 5 innovative restaurant concepts."}
    ],
    temperature=0.9,
    top_p=0.95
)

The Practical Rule: Adjust One Parameter at a Time

The Mistral documentation recommends a methodical approach that you should adopt from your first tests: set Top P to 1.0 (no filtering) and adjust only temperature, or set temperature to 1.0 and adjust only Top P; only then do you refine the second parameter if needed. The reason is simple: if you change both simultaneously and the behavior changes, you won’t know which one is responsible. It’s the same reflex as debugging—you change only one variable at a time.

To give you concrete starting points: data extraction or JSON output is typically set to temperature=0.0, top_p=1.0; professional writing to temperature=0.4, top_p=0.9; creative generation to temperature=0.8, top_p=0.95; and code or technical content to temperature=0.2, top_p=0.9. These values are just starting points—experimentation remains essential for your specific use case.

Key Takeaways

• Top P filters candidate tokens by cumulative probability threshold
• Unlike temperature which softens the distribution, Top P cuts it
• Adjust one parameter at a time to understand its impact
• Low Top P = focused responses, High Top P = varied responses
• Start with temperature alone, add Top P to refine