Moderation Scores and Thresholds

Updated November 3, 2026. Threshold profiles previously referenced non-existent categories (hate, violence, unpredictable); they now align with the eleven keys from mistral-moderation-2603, including dangerous. The score interpretation guide is provided as a training reference, as Mistral does not publish one. The monitor previously raised a KeyError for any category missing from the profile and used datetime.utcnow(), which has been deprecated since Python 3.12. Sources: Moderation & Guardrailing · datetime.utcnow.

Interpreting Moderation Scores

Each category returns a score between 0.0 and 1.0. But what do these numbers actually mean? And how do you set decision thresholds for your application? This lesson gives you the keys to turn raw scores into automated decisions—moving from measurement to policy.

First, understand that Mistral does not present these scores as calibrated probabilities. A score of 0.6 does not mean “a six in ten chance this text is dangerous”; it is a strong signal from the classifier. The documentation highlights two useful points. The boolean categories in the response apply thresholds Mistral chose for optimal performance on its internal test set; you can use them, or start from the raw scores and set your own. On that test set, category precision ranges from 0.8 to 1.0, and recall from 0.7 to 1.0: false positives and missed content both occur, which is why manually tuned thresholds matter.

The grid below is not published by Mistral: it is a starting reference to validate against your own metrics.

Defining Your Thresholds: The Decision Matrix

Choosing a threshold is a trade-off between safety (blocking as much as possible) and usability (minimizing false positives). No setting wins on both fronts, and claiming otherwise leads to endless back-and-forth. The working method is to define three explicit profiles and map each application to one of them.

The strict profile targets a consumer application where minors may be present; the balanced profile fits a B2B professional application with identified users; the permissive profile is reserved for an internal tool handled by experts who know what they are asking for.

# Three threshold profiles
PROFILES = {
    "strict": {
        # Consumer application, children possible
        "sexual": 0.1,
        "hate_and_discrimination": 0.1,
        "violence_and_threats": 0.1,
        "dangerous": 0.1,
        "criminal": 0.1,
        "selfharm": 0.1,
        "health": 0.2,
        "financial": 0.2,
        "law": 0.2,
        "pii": 0.1,
        "jailbreaking": 0.1,
    },
    "balanced": {
        # B2B professional application
        "sexual": 0.2,
        "hate_and_discrimination": 0.15,
        "violence_and_threats": 0.25,
        "dangerous": 0.15,
        "criminal": 0.15,
        "selfharm": 0.15,
        "health": 0.4,
        "financial": 0.4,
        "law": 0.4,
        "pii": 0.2,
        "jailbreaking": 0.15,
    },
    "permissive": {
        # Internal tool for experts
        "sexual": 0.3,
        "hate_and_discrimination": 0.2,
        "violence_and_threats": 0.4,
        "dangerous": 0.2,
        "criminal": 0.2,
        "selfharm": 0.2,
        "health": 0.7,
        "financial": 0.7,
        "law": 0.7,
        "pii": 0.3,
        "jailbreaking": 0.2,
    },
}

Compare the health column across profiles: 0.2, then 0.4, then 0.7. The same exchange about dosage will be blocked in the first case and accepted in the last. Usage context decides, not the nature of the text.

Multi-Level Classification System

Blocking or accepting is an oversimplified view of reality. Between manifestly safe content and manifestly violating content lies a gray zone where the right decision is to display while flagging. The function below introduces this third state: above the threshold, block; above 60% of the threshold, warn and log; below that, let it pass.

from enum import Enum

class RiskLevel(Enum):
    SAFE = "safe"          # Green: safe content
    WARNING = "warning"    # Orange: content to monitor
    BLOCKED = "blocked"    # Red: blocked content

def classify_content(scores: dict, thresholds: dict) -> dict:
    """Classifies content into 3 risk levels."""
    blocking_violations = []
    warnings = []

    for category, threshold in thresholds.items():
        score = scores.get(category, 0)

        if score > threshold:
            blocking_violations.append({
                "category": category,
                "score": round(score, 3)
            })
        elif score > threshold * 0.6:
            # Warning zone: 60% of threshold
            warnings.append({
                "category": category,
                "score": round(score, 3)
            })

    if blocking_violations:
        return {
            "level": RiskLevel.BLOCKED,
            "violations": blocking_violations,
            "action": "block_and_log"
        }
    elif warnings:
        return {
            "level": RiskLevel.WARNING,
            "warnings": warnings,
            "action": "display_and_log"
        }
    else:
        return {
            "level": RiskLevel.SAFE,
            "action": "display"
        }

The value of the orange zone is as much operational as it is educational: it shows you what is brushing against your thresholds. If a category generates many warnings without ever crossing the line, you know that an adjustment of a few hundredths would shift a significant volume of traffic.

Worked Example: Analyze and Decide

The assembly comes down to one function: call the classifier, apply the profile, make the decision. The comparison below serves as a demonstration—a paracetamol dosage is blocked in strict profile under health and accepted in permissive profile, without a single line of text changing.

from mistralai.client import Mistral
import os

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])

def moderate_and_decide(text: str, profile: str = "balanced") -> dict:
    """Moderates text and makes a decision based on the profile."""
    # Call to moderation API
    response = client.classifiers.moderate(
        model="mistral-moderation-2603",
        inputs=[text]
    )
    scores = response.results[0].category_scores or {}

    # Apply profile thresholds
    thresholds = PROFILES[profile]
    classification = classify_content(scores, thresholds)

    return {
        "text": text[:80],
        "profile": profile,
        "raw_scores": {k: round(v, 3) for k, v in scores.items()},
        "decision": classification
    }

# Test with different profiles
text = "Take 2 paracetamol tablets every 6 hours"

# Strict profile: likely blocked (health)
result_strict = moderate_and_decide(text, "strict")

# Permissive profile: likely accepted
result_permissive = moderate_and_decide(text, "permissive")

Adjusting Thresholds Dynamically

In production, your thresholds must evolve, and you steer that evolution with data. The monitor below writes every decision to a JSONL file and maintains counters for blocks, warnings, and passes, category by category. The resulting report gives you a block rate per category—the only metric that tells you if a threshold is too low.

import json
from datetime import datetime, timezone

class ModerationMonitor:
    """Collects moderation metrics to adjust thresholds."""

    def __init__(self, log_file: str = "moderation_logs.jsonl"):
        self.log_file = log_file
        self.stats = {cat: {"blocked": 0, "warned": 0, "passed": 0}
                     for cat in PROFILES["balanced"].keys()}

    def log_decision(self, scores: dict, thresholds: dict, decision: str):
        """Logs a moderation decision."""
        entry = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "scores": scores,
            "thresholds": thresholds,
            "decision": decision
        }
        with open(self.log_file, "a") as f:
            f.write(json.dumps(entry) + "\n")

        # Update counters
        for cat, threshold in thresholds.items():
            score = scores.get(cat, 0.0)
            self.stats.setdefault(cat, {"blocked": 0, "warned": 0, "passed": 0})
            if score > threshold:
                self.stats[cat]["blocked"] += 1
            elif score > threshold * 0.6:
                self.stats[cat]["warned"] += 1
            else:
                self.stats[cat]["passed"] += 1

    def report(self) -> dict:
        """Generates a report to adjust thresholds."""
        report = {}
        for cat, stats in self.stats.items():
            total = sum(stats.values())
            if total > 0:
                report[cat] = {
                    "block_rate": round(stats["blocked"] / total, 3),
                    "warning_rate": round(stats["warned"] / total, 3),
                    "total": total
                }
        return report

Pitfalls to Avoid

The first pitfall is a single threshold applied to all categories. Each category has its own score distribution: a threshold of 0.3 for hate_and_discrimination does not mean the same thing as 0.3 for health, where legitimate content naturally lives higher on the scale. Adjust each category individually, even if that means making eleven decisions instead of one.

The second is ignoring false positives. A high block rate erodes user trust much faster than an isolated incident, because it affects people acting in good faith. Track the block rate by category and raise thresholds when the data justifies it.

The third, more insidious, is failing to log decisions. Without logs, you cannot adjust your thresholds in an informed way and you revert to opinion-based discussions. Systematically record scores, applied thresholds, and decisions made—this corpus is what will make the next revision fast and indisputable.

Key Takeaways

• Scores range from 0.0 to 1.0; they are not calibrated probabilities, and Mistral encourages setting your own thresholds
• Define threshold profiles adapted to your context (strict, balanced, permissive)
• Implement a 3-level system: safe, warning, blocked
• Log all decisions to continuously adjust your thresholds
• Each category requires its own threshold—no single value fits all