Define Your Content Policy — AI Moderation and Security with Mistral

Updated November 3, 2026. Previous policies used three non-existent categories (hate, violence, unpredictable) and omitted dangerous. They now align with the eleven keys from mistral-moderation-2603. A method now converts a policy into the official guardrails parameter format, ensuring consistent thresholds are applied before generation. Sources: Moderation & Guardrailing · mistralai SDK 3.7.0.

Why You Need a Formal Content Policy

Moderation thresholds aren’t arbitrary technical choices. They reflect a content policy—a set of rules defining what your application allows and forbids. Without a formal policy, every developer makes their own calls, and your security system loses consistency: two services within the same company end up blocking different content, with no one knowing why.

A written policy lets you justify moderation decisions to both users and regulators, harmonize thresholds across teams and applications, audit system compliance, and iterate in a structured way as needs change. The day a user challenges a block, the difference between “the threshold is set to 0.1” and “our Public-Facing Policy v1.2 sets this threshold for this reason” is massive.

Tailoring Thresholds to Application Context

The same content might be acceptable in one context and dangerous in another. Your policy must reflect these nuances. The best way to achieve this is to describe each profile as a named, dated, and documented object, rather than an anonymous dictionary lost in the code.

For a public-facing chatbot, the rule is simple: strict everywhere, with safety as the priority. You know nothing about the person on the other end, and minors may be present.

# Policy: Strict on everything, safety first
POLICY_PUBLIC = {
    "name": "Public-Facing",
    "description": "Chatbot accessible to all, including minors",
    "thresholds": {
        "sexual": 0.1,
        "hate_and_discrimination": 0.1,
        "violence_and_threats": 0.1,
        "dangerous": 0.1,
        "criminal": 0.1,
        "selfharm": 0.1,
        "health": 0.2,
        "financial": 0.2,
        "law": 0.2,
        "pii": 0.1,
        "jailbreaking": 0.1,
    },
    "block_action": "generic_message",
    "logging": "full"
}

An internal assistant changes the game on one point: users are authenticated, meaning they are identifiable and accountable. This allows for more flexible thresholds, particularly on financial and law set to 0.6, since an employee querying the assistant about a contract structure is simply doing their job.

# Policy: Balanced, identified users
POLICY_INTERNAL = {
    "name": "Internal B2B",
    "description": "Assistant for authenticated employees",
    "thresholds": {
        "sexual": 0.2,
        "hate_and_discrimination": 0.15,
        "violence_and_threats": 0.3,
        "dangerous": 0.2,
        "criminal": 0.15,
        "selfharm": 0.15,
        "health": 0.5,
        "financial": 0.6,
        "law": 0.6,
        "pii": 0.3,
        "jailbreaking": 0.15,
    },
    "block_action": "detailed_message",
    "logging": "full"
}

A specialized tool—medical, legal, or financial—takes this logic to its conclusion: very permissive within its domain, strict everywhere else. In the medical example, health rises to 0.85 because discussing diagnoses is precisely the tool’s function, while pii drops to 0.15 because patient data must never circulate. Two elements absent from previous policies are added here: a blocking mode with redirection and an explicit disclaimer.

# Policy: Permissive on domain, strict elsewhere
POLICY_MEDICAL = {
    "name": "Medical Assistant",
    "description": "Tool for healthcare professionals",
    "thresholds": {
        "sexual": 0.2,
        "hate_and_discrimination": 0.1,
        "violence_and_threats": 0.3,
        "dangerous": 0.1,
        "criminal": 0.1,
        "selfharm": 0.3,
        "health": 0.85,    # Very permissive: legitimate domain
        "financial": 0.3,
        "law": 0.4,
        "pii": 0.15,       # Strict: patient data
        "jailbreaking": 0.1,
    },
    "block_action": "detailed_message_with_redirect",
    "logging": "full_with_audit",
    "disclaimer": "This tool does not replace medical advice."
}

Formalizing the Policy as a Document

Your content policy must be a versioned document accessible to all teams. In Python, this translates to a class that carries the version number and revision date alongside the thresholds, and that can both evaluate scores and generate the blocking message corresponding to the declared mode.

from mistralai.client.models import GuardrailConfigTypedDict


class ContentPolicy:
    """Structured representation of a content policy."""

    def __init__(self, config: dict):
        self.name = config["name"]
        self.description = config["description"]
        self.thresholds = config["thresholds"]
        self.block_action = config["block_action"]
        self.logging = config["logging"]
        self.version = config.get("version", "1.0.0")
        self.revision_date = config.get("revision_date", "2026-04-01")

    def evaluate(self, scores: dict) -> dict:
        """Evaluates scores against the policy."""
        violations = []
        for cat, threshold in self.thresholds.items():
            score = scores.get(cat, 0)
            if score > threshold:
                violations.append({
                    "category": cat,
                    "score": round(score, 3),
                    "threshold": threshold
                })

        return {
            "policy": self.name,
            "version": self.version,
            "allowed": len(violations) == 0,
            "violations": violations
        }

    def block_message(self, violations: list) -> str:
        """Generates an appropriate blocking message."""
        if self.block_action == "generic_message":
            return "This content cannot be processed."
        elif self.block_action == "detailed_message":
            cats = ", ".join(v["category"] for v in violations)
            return (f"Content blocked for the following reasons: "
                    f"{cats}. Please rephrase your request.")
        return "Content blocked."

    def to_guardrails(self) -> list[GuardrailConfigTypedDict]:
        """Translates the policy into the guardrails parameter for the Mistral API."""
        return [
            {
                "block_on_error": True,
                "moderation_llm_v2": {
                    "custom_category_thresholds": self.thresholds,
                    "ignore_other_categories": False,
                    "action": "block",
                },
            }
        ]

The to_guardrails method makes the policy the single source of truth for guardrails: client.chat.complete(..., guardrails=policy.to_guardrails()) or client.beta.agents.create(..., guardrails=policy.to_guardrails()). Thresholds applied before and after generation can no longer diverge.

Note also that the result of evaluate includes the name and version of the applied policy. This detail allows you, six months later, to replay a contested decision knowing exactly under which regime it was made.

Multi-Application Management

If your company manages multiple applications, centralize policy management. The manager below does two useful things: it explicitly refuses to serve an application without a declared policy, preventing deployments without safety nets, and it produces a compliance report listing each application’s version and last revision date—exactly what an auditor will ask for.

class PolicyManager:
    """Manages content policies for multiple applications."""

    def __init__(self):
        self.policies = {}

    def register(self, app_id: str, policy: ContentPolicy):
        """Registers a policy for an application."""
        self.policies[app_id] = policy

    def get(self, app_id: str) -> ContentPolicy:
        """Retrieves the policy for an application."""
        if app_id not in self.policies:
            raise ValueError(f"No policy for {app_id}")
        return self.policies[app_id]

    def compliance_report(self) -> dict:
        """Generates a global compliance report."""
        return {
            app_id: {
                "name": pol.name,
                "version": pol.version,
                "revision_date": pol.revision_date,
                "categories_covered": len(pol.thresholds)
            }
            for app_id, pol in self.policies.items()
        }

# Usage
manager = PolicyManager()
manager.register("chatbot-public",
    ContentPolicy(POLICY_PUBLIC))
manager.register("assistant-medical",
    ContentPolicy(POLICY_MEDICAL))

Revision and Updates

Your policy isn’t set in stone. The most common mistake is writing it once and then forgetting it. Establish a rhythm: every week, analyze false positives and false negatives reported in your logs; every month, adjust thresholds based on accumulated metrics; every quarter, conduct a full revision with business, legal, and product stakeholders. Add to this a non-negotiable rule: after every incident, perform a post-mortem and implement an immediate fix, without waiting for the next scheduled review.

Key Takeaways

A content policy translates your technical thresholds into business rules
Tailor thresholds to the context: public-facing, internal, specialized
Formalize the policy in a versioned document
Centralize management if you have multiple applications
Revise regularly—the policy must evolve with your product