Offline evaluations: measure before you deploy

Lesson rewritten on November 2, 2026 based on Mistral’s Offline evaluations documentation: the “Campaigns” from the previous version and their campaigns.* methods no longer exist. Large-scale evaluation now uses the mistralai-evaluations SDK, which runs a task on a dataset, scores it with evaluators, and publishes the results in Studio. Sources: Offline evaluations · Evaluators · Goals. Update on November 3: mistralai-evaluations 0.9.1 requires mistralai < 3 and replaces the 3.x SDK with a 2.x version when installed alongside it; the lesson recommends using a dedicated environment (dependencies on PyPI).

Moving from single cases to scale

A judge validated on twenty conversations tells you nothing about your application as a whole. To decide whether a new prompt, model, or setting is better than the current one, you must measure it against a representative, consistent set of cases and compare the results. This is the role of offline evaluations.

These are reserved for Enterprise organizations and require Python 3.12 or later:

pip install mistralai-evaluations

Install it in a separate environment. Version 0.9.1, the latest as of November 3, 2026, requires mistralai 2.x (mistralai>=2.7.1,<3). In an environment that already contains the 3.x SDK, pip uninstalls it and installs a 2.x version, with no warning other than its output lines: verified with mistralai 3.7.0, replaced by 2.10.1. A dedicated virtual environment for evaluations prevents regressions in your application.

The mental model

The SDK follows a simple flow:

Dataset  →  Task  →  Evaluators  →  Results

• Dataset: a list of records (dictionaries) or a Studio dataset
• Task: a function, preferably asynchronous, that receives a TaskContext and produces an output—this is the code you are evaluating
• Evaluators: functions that receive a ScorerContext and score each output
• Results: statistics and scores per record, displayed in the terminal and published in Studio

A first evaluation

import asyncio
import os

from mistralai.evaluations import (
    Evaluation, Evaluator, Goal, Mistral, Project, ScorerContext, System, TaskContext,
)

client = Mistral(api_key=os.environ["MISTRAL_API_KEY"])

dataset = [
    {"ticket": "I haven't been able to log in since this morning.", "categorie": "acces"},
    {"ticket": "My October invoice charges the same subscription twice.", "categorie": "facturation"},
    {"ticket": "The PDF export crashes as soon as the report exceeds 50 pages.", "categorie": "bug"},
]


async def tache(ctx: TaskContext) -> str:
    response = await client.chat.complete_async(
        model=str(ctx.system.params["model"]),
        messages=[
            {"role": "system", "content": str(ctx.system.params["system_prompt"])},
            {"role": "user", "content": ctx.input_record["ticket"]},
        ],
    )
    return str(response.choices[0].message.content)


def exactitude(ctx: ScorerContext) -> int:
    return 1 if ctx.input_record["categorie"] == str(ctx.output).strip().lower() else 0


async def main():
    run = await client.evaluation.run(
        project=Project(name="Tri des tickets"),
        evaluation=Evaluation(name="Exactitude du classement"),
        system=System(name="small-prompt-v1", params={
            "model": "mistral-small-latest",
            "system_prompt": "Classe le ticket : acces, facturation ou bug. Réponds par le seul mot.",
        }),
        dataset=dataset,
        task=tache,
        evaluators=[
            Evaluator(
                name="exactitude",
                description="1 si la catégorie prédite est la bonne, 0 sinon.",
                scorer=exactitude,
                goal=Goal.gte(0.8),
            ),
        ],
    )
    run.show(level="records")


asyncio.run(main())

Results appear in the terminal and are published in Studio under Observability > Evaluate > Evaluations. The SDK calculates the mean, minimum, maximum, and standard deviation for each evaluator.

Prefer an asynchronous task for long-running processes: a synchronous task runs on the event loop that also sends heartbeat signals for the run to Studio, and a call blocked for several minutes can cause the run to be marked as canceled.

Two types of evaluators

An evaluator associates a name, an optional description, a scoring function, and, if needed, a goal. The function can be:

• Rule-based: a direct calculation on the output—exact, fast, and free;
• An LLM judge: a function that calls a model to score what is hard to code—utility, factual accuracy, tone.

JUDGE_PROMPT = """Tu es un correcteur exigeant. Note de 0 à 5 l'utilité de la réponse
pour la request ci-dessous. Réponds par le seul chiffre.

Demande : {request}
Réponse : {response}
"""


def judge_utility(ctx: ScorerContext) -> int:
    response = client.chat.complete(
        model="mistral-large-latest",
        messages=[{"role": "user", "content": JUDGE_PROMPT.format(
            request=ctx.input_record["ticket"], response=ctx.output)}],
        temperature=0,
    )
    raw = str(response.choices[0].message.content).strip()
    return int(raw[0]) if raw[:1].isdigit() else 0

You can freely mix both types in a single run: pass them together in the evaluators list. An evaluator can also return a string or a boolean; the SDK then produces a frequency distribution rather than averages.

Comparing versions

The System object captures the task configuration: model, temperature, system prompt, or any other setting, in a free-form dictionary. Each run records its System, and Studio allows you to compare runs side by side. To decide between two prompts or two models, rerun the same evaluation with two different System objects, without touching the task code.

Setting goals

A goal turns a score into a PASS or FAIL verdict: Goal.gte(0.8) requires a mean of at least 0.8, Goal.lte(...) sets a ceiling, and Goal.between(...) defines a range. Each goal targets the mean by default, or another statistic with metric ("min", "max", "std"): Goal.gte(0.8, metric="min") requires every record to reach 0.8. This is what makes an evaluation a guardrail: a new version that drops a score below its goal does not go to production.

Key takeaways

• Offline evaluations (mistralai-evaluations, Python 3.12+, Enterprise organizations) replace the old Campaigns
• Dataset, task, evaluators, results: client.evaluation.run(...) chains it all and publishes to Studio
• Evaluators are functions: exact rules or LLM judges, freely combined
• System records the evaluated configuration and enables version comparison
• Goals (Goal.gte, Goal.lte, Goal.between) provide PASS or FAIL verdicts