Presence and Frequency Penalties

Combating Repetition

Language models have a natural tendency to repeat themselves. This shows up in three recognizable ways: the same word looping through a paragraph, identical sentence structures repeating line after line, or entire paragraphs rephrased without adding new information. Temperature doesn’t fix this—cranking up global randomness degrades coherence without guaranteeing lexical variety. The presence_penalty and frequency_penalty parameters offer much finer control, provided you understand what sets them apart.

Presence Penalty: A Fixed Penalty

The presence penalty applies a single, fixed penalty to every token that has appeared at least once in the generated response. It doesn’t matter whether the token was used once or fifteen times—the penalty stays the same. The effect is to encourage the model to draw from words it hasn’t used yet, which broadens the response’s vocabulary. Ask for ten adjectives to describe the sea and you’ll see the difference immediately—without a penalty, variations tend to circle around three or four semantic fields.

from mistralai.client import Mistral
import os

client = Mistral(api_key=os.getenv("MISTRAL_API_KEY"))

# No penalty — the model may repeat itself
response = client.chat.complete(
    model="mistral-large-latest",
    messages=[{"role": "user", "content": "List 10 adjectives to describe the sea."}],
    presence_penalty=0.0
)
print("No penalty:", response.choices[0].message.content)

# With presence penalty — encourages vocabulary diversity
response = client.chat.complete(
    model="mistral-large-latest",
    messages=[{"role": "user", "content": "List 10 adjectives to describe the sea."}],
    presence_penalty=1.0
)
print("With penalty:", response.choices[0].message.content)

The parameter accepts values from -2.0 to 2.0, with 0.0 as the default. A positive value penalizes tokens already seen and therefore produces more diversity; a negative value does the opposite and favors repetition, which is useful only in very specific cases like generating choruses or repeated formulas. In production, stay within the 0.0 to 1.5 range.

Frequency Penalty: A Cumulative Penalty

The frequency penalty works differently: the penalty is cumulative and grows with the number of occurrences of the token. The more a word is repeated, the less likely it becomes on the next occurrence. Where presence penalty opens up the vocabulary, this one progressively slows down words that come back too often, making it particularly effective against loops in long texts.

# No frequency penalty
response = client.chat.complete(
    model="mistral-large-latest",
    messages=[{"role": "user", "content": "Write a paragraph about artificial intelligence."}],
    frequency_penalty=0.0
)
print("No penalty:", response.choices[0].message.content)

# With frequency penalty — reduces very frequent tokens
response = client.chat.complete(
    model="mistral-large-latest",
    messages=[{"role": "user", "content": "Write a paragraph about artificial intelligence."}],
    frequency_penalty=0.8
)
print("With penalty:", response.choices[0].message.content)

Like the previous one, it ranges from -2.0 to 2.0 with 0.0 as the default, with a positive value increasingly penalizing frequent tokens. Its useful range in production is narrower, however: 0.0 to 1.0, because its cumulative nature makes it more aggressive at equal values.

The following table summarizes what sets them apart.

Configurations by Intent

In practice, you tune these two parameters alongside temperature, depending on what your application produces. A chatbot needs slight diversity and no repetitions, nothing more. Creative writing pushes presence penalty to enrich vocabulary while keeping frequency penalty moderate. Data extraction, on the other hand, disables both: repetition is legitimate there, and the same field name may need to appear ten times. Summarization sits in between, with just enough penalty to avoid copy-pasting the source text.

# Chatbot — slight diversity, no repetitions
CONFIG_CHATBOT = {
    "temperature": 0.7,
    "presence_penalty": 0.3,
    "frequency_penalty": 0.3
}

# Creative writing — rich vocabulary, no loops
CONFIG_CREATIF = {
    "temperature": 0.9,
    "presence_penalty": 0.8,
    "frequency_penalty": 0.5
}

# Data extraction — no penalties (exact response)
CONFIG_EXTRACTION = {
    "temperature": 0.0,
    "presence_penalty": 0.0,
    "frequency_penalty": 0.0
}

# Summary — some diversity to avoid copy-paste
CONFIG_RESUME = {
    "temperature": 0.3,
    "presence_penalty": 0.2,
    "frequency_penalty": 0.4
}

Diagnose Before You Tune

Tuning these parameters by ear quickly leads to arbitrary adjustments. Measure instead. The function below calculates a diversity ratio—unique words over total words—and isolates words longer than three letters that appear more than twice, which filters out articles and prepositions whose repetition is normal. Run it on the same prompt with three frequency_penalty values and you’ll see the ratio climb, then plateau: that plateau marks the value beyond which you gain nothing more.

def analyze_repetition(text: str) -> dict:
    """Analyzes the repetition rate of generated text."""
    words = text.lower().split()
    total = len(words)
    unique = len(set(words))

    # Count words that appear more than 2 times
    from collections import Counter
    counts = Counter(words)
    repeated = {w: c for w, c in counts.items() if c > 2 and len(w) > 3}

    return {
        "total_words": total,
        "unique_words": unique,
        "diversity_ratio": round(unique / total, 3) if total > 0 else 0,
        "repeated_words": dict(sorted(repeated.items(), key=lambda x: -x[1])[:10])
    }

# Test with and without penalties
for fp in [0.0, 0.5, 1.0]:
    response = client.chat.complete(
        model="mistral-large-latest",
        messages=[{"role": "user", "content": "Write a long paragraph about renewable energy."}],
        max_tokens=500,
        frequency_penalty=fp
    )
    text = response.choices[0].message.content
    stats = analyze_repetition(text)
    print(f"frequency_penalty={fp} | Diversity: {stats['diversity_ratio']} | Repeated: {stats['repeated_words']}")

Two Ways to Ruin Everything

The first is going too high. Beyond 1.5, the model avoids common words so much that it substitutes inappropriate alternatives: the text becomes disjointed, riddled with misused rare terms. In an explanation of the Pythagorean theorem, penalties at 2.0 prevent the model from repeating “side” and “triangle”—precisely the words the proof needs.

# BAD — penalty too strong
response = client.chat.complete(
    model="mistral-large-latest",
    messages=[{"role": "user", "content": "Explain the Pythagorean theorem."}],
    presence_penalty=2.0,
    frequency_penalty=2.0
)
# Result: disjointed text, misused rare words

The second is combining high penalties with high temperature. Each mechanism adds its share of randomness to the same selection process, and their superposition produces nearly random text. As with temperature and top_p, the discipline is to adjust one parameter at a time.

# AVOID — too much randomness
{"temperature": 1.5, "presence_penalty": 1.5, "frequency_penalty": 1.5}

# CORRECT — adjust one parameter at a time
{"temperature": 0.7, "presence_penalty": 0.5, "frequency_penalty": 0.3}

Key Takeaways

• Presence penalty: fixed penalty once a token is seen — broadens vocabulary
• Frequency penalty: cumulative penalty proportional to frequency — combats loops
• Both range from -2.0 to 2.0, default 0.0
• Stay in the 0.0 to 1.0 range in production for most cases
• Don’t combine high values with high temperature
• Disable penalties for extraction and classification (where repetition is acceptable)