Datasets: Build evaluation sets and iterate

Lesson rewritten on November 2, 2026 based on the Mistral Datasets page and mistralai SDK 3.7.0: a record contains a free-form payload and properties; datasets are populated via manual entry, file import, or spans from the Explorer, and are passed as-is to offline evaluations. The import_from_explorer method from the previous version no longer exists. Sources: Datasets · Explore traces.

The final link in the iteration loop

A dataset is the collection of test cases that drives an offline evaluation. It gives your quality process memory: every incident observed in production can become a test case, and every new version of the application is measured against it. Without a stable dataset, you’re comparing impressions; with one, you’re comparing numbers.

Record structure

In Studio, each record contains a payload—a JSON object where you define the keys—and properties, which are metadata not passed to the evaluated task. During an evaluation, the payload becomes the input record exactly as is (ctx.input_record).

{"payload": {"ticket": "My October invoice lists the same subscription twice.", "category": "billing"}, "properties": {"source": "incident-2026-10-14"}}

This line also represents the JSONL import format for Studio datasets: one object per line, containing payload and properties.

Data sources

You can populate a dataset in several ways:

• Manual entry, in Studio or via the API, record by record
• JSONL file import using the format above
• From the Explorer: a single span, or up to 500 selected spans, become records (Lesson 12)
• From the Studio Playground or from another dataset

from mistralai.client import Mistral

client = Mistral(api_key="your-api-key")
datasets = client.beta.observability.datasets

# Create the dataset
dataset = datasets.create(
    name="Support tickets — reference set",
    description="Manually validated ticket classification cases",
)
print(f"Dataset: {dataset.id} ({dataset.slug})")

# Add records
cases = [
    {"ticket": "I haven't been able to log in since this morning.", "category": "access"},
    {"ticket": "The PDF export crashes whenever the report exceeds 50 pages.", "category": "bug"},
]
for payload in cases:
    datasets.create_record(
        dataset_id=dataset.id,
        payload=payload,
        properties={"origin": "manual entry"},
    )

For large volumes, import_from_file imports an already uploaded JSONL file (by its file_id), and import_from_spans ingests trace spans: these imports run as background tasks, and you can track their progress with fetch_task.

Review and export

list_records iterates through records page by page (page_size, page), and export_to_jsonl generates an export whose response provides the download URL (file_url):

page = datasets.list_records(dataset_id=dataset.id, page_size=50, page=1)
export = datasets.export_to_jsonl(dataset_id=dataset.id)
print(export.file_url)

Evaluate on a Studio dataset

An offline evaluation reads a Studio dataset by its slug or ID:

from mistralai.evaluations import Dataset, Evaluation, Evaluator, Project

run = await client.evaluation.run(
    project=Project(name="Ticket sorting"),
    evaluation=Evaluation(name="Reference set"),
    dataset=Dataset(slug="support-tickets-reference-set"),
    task=tache,
    evaluators=[Evaluator(name="exactitude", scorer=exactitude)],
)

Here, client is the evaluation SDK client, and tache and exactitude are defined in Lesson 14. Pass exactly one of slug or id; selecting a dataset by its slug never creates a new one. Keep two rules in mind: a run reads the records present at startup, up to 10,000—a dataset is not a snapshot, so modifying it only affects subsequent runs—and if records are added or removed during the read, the run fails before starting rather than executing on an inconsistent set.

Management best practices

Keep a stable reference set. A dataset that changes with every run makes comparison impossible. Create a reference set that you only modify deliberately, and maintain separate working datasets alongside it.

Feed it from production. The most useful cases are those that actually caused problems: a span spotted in the Explorer, where you correct the expected output, is worth more than ten invented cases.

Document provenance. The properties (origin, date, related incident) aren’t seen by the task, but they’ll tell you six months from now why a case exists.

Balance your data. A dataset containing only easy cases flatters your system; one containing only traps discourages it. Aim for the actual distribution of your traffic, plus a handful of acknowledged edge cases.

Key takeaways

• A dataset record = a payload (task input) and properties (metadata)
• Manual entry, JSONL import, Explorer spans, Playground: multiple sources for a single dataset
• The SDK exposes create, create_record, import_from_file, import_from_spans, list_records, and export_to_jsonl
• Evaluations read a dataset via Dataset(slug=...) or Dataset(id=...), up to 10,000 records
• A dataset is not a snapshot: keep a stable reference set to compare versions