Judges: Automated LLM Evaluation — Production: Batch, Optimization, and Observability

Lesson updated November 2, 2026 based on mistralai SDK 3.7.0 and the Mistral Evaluators page: judges.create now requires description and tools (older examples would fail), regression outputs require min_description and max_description, and you can test a judge directly on a conversation using judge_conversation. Template variables and “Campaigns” from earlier versions no longer appear in the documentation. Sources: Evaluators · client-python.

Automating Quality Evaluation

Manually reviewing thousands of conversations is impossible at production scale. Judges solve this problem: they are LLM-based evaluators that you configure once—defining instructions, model, and output format—and then apply to conversations or dataset records. Each judgment produces an analysis and a verdict: a label or a score.

Two Types of Judges

Classification

A classification judge assigns discrete labels to each response. Examples include:

Binary: helpful / not_helpful
Multi-class: excellent / acceptable / poor
Categorization: code_question / general_knowledge / troubleshooting
Safety: safe / needs_review / unsafe

Regression

A regression judge assigns a numerical score within a range, where you define the boundaries:

Helpfulness rating from 1 to 5
Relevance score from 0 to 100
Confidence level from 0.0 to 1.0

Writing Effective Instructions

The judge’s instructions determine the entire quality of the evaluation, and vagueness is the primary cause of failure. “Evaluate the quality of the response” means nothing to a model—each run will interpret “quality” differently. A working version names observable criteria: “Evaluate whether the response directly answers the question asked, using factually correct information.” The same principle applies to context: your judge doesn’t know it’s evaluating a support chatbot where “good” means “resolves the customer’s issue without sending them to the documentation.” If you don’t state this explicitly, it will apply a generic definition of quality that isn’t yours.

Edge cases deserve special attention, as this is where evaluations diverge. Specifying that “a score of 3 means the response is partially correct but missing a key element” stabilizes the middle of your scale—without this anchor, the same conversation might receive a 2, 3, or 4 depending on the run. The ultimate test is simple: if two human evaluators reading your instructions could disagree on a given case, your criterion is too vague, and the judge will inherit that ambiguity at scale.

Creating a Classification Judge via the SDK

from mistralai.client import Mistral

client = Mistral(api_key="your-api-key")

# Three-level classification judge
classifier = client.beta.observability.judges.create(
    name="Support Response Quality",
    description="Classifies each support chatbot response into three quality levels.",
    model_name="mistral-medium-latest",
    instructions="""
Evaluate the assistant's last response in the provided conversation.

Context: This is a technical support chatbot.

Criteria:
- Is the response factual and correct?
- Does it directly answer the question asked?
- Is the tone professional and empathetic?

Classify the response according to the proposed categories.
""",
    output={
        "type": "CLASSIFICATION",
        "options": [
            {"value": "excellent", "description": "Correct, complete, and professional response"},
            {"value": "acceptable", "description": "Correct but incomplete response, or tone needs improvement"},
            {"value": "poor", "description": "Incorrect, off-topic, or unprofessional response"},
        ],
    },
    tools=[],  # No tools for this judge
)

print(f"Judge created: {classifier.id}")

Creating a Regression Judge via the SDK

# Regression judge: Helpfulness score from 1 to 5
scorer = client.beta.observability.judges.create(
    name="Helpfulness Score",
    description="Rates the helpfulness of the assistant's last response on a scale of 1 to 5.",
    model_name="mistral-small-latest",
    instructions="""
Evaluate the helpfulness of the assistant's last response on a scale of 1 to 5.

1 = Useless (off-topic, incorrect, or does not answer the question)
2 = Slightly helpful (partially correct but misses the main point)
3 = Moderately helpful (correct but incomplete or too vague)
4 = Helpful (correct and sufficiently detailed)
5 = Very helpful (correct, detailed, with clear examples or steps)
""",
    output={
        "type": "REGRESSION",
        "min": 1,
        "max": 5,
        "min_description": "Useless response: off-topic, incorrect, or no answer provided",
        "max_description": "Very helpful response: correct, detailed, with clear steps",
    },
    tools=[],
)

print(f"Judge created: {scorer.id}")

Validate Before Scaling

An unvalidated judge produces false annotations at scale. Before relying on one, test it on conversations where you know the correct answer: judge_conversation applies it to a list of messages and returns its analysis and verdict.

result = client.beta.observability.judges.judge_conversation(
    judge_id=classifier.id,
    messages=[
        {"role": "user", "content": "My printer shows error E04, what should I do?"},
        {"role": "assistant", "content": "Check the documentation."},
    ],
)

print(result.analysis)  # The judge's reasoning
print(result.answer)    # The label or score

Then proceed methodically:

Test on 10 to 20 cases where you have predetermined the correct verdict
Check agreement: Does the judge produce the same results as you?
Test stability: When rerunning on the same cases, are the results identical?
Identify failure patterns: On what types of conversations does the judge make mistakes?

If agreement is too low, refine the instructions before any large-scale use. Once validated, a judge applies to dataset records (client.beta.observability.datasets.records.judge), and the same LLM judge logic appears in the evaluators for offline evaluations (Lesson 14).

Key Takeaways

An LLM judge classifies (labels) or scores (rates) conversations, supported by an analysis
judges.create requires name, description, model_name, instructions, output, and tools
A regression output defines its boundaries (min_description, max_description)
Instructions must be specific, contextualized, and anchored with score examples
Validate a judge with judge_conversation on known cases before relying on it