End-to-end pipeline: traces, judges, datasets, evaluations

Lesson rewritten on November 2, 2026 based on Mistral’s Observability documentation and mistralai SDK 3.7.0: the pipeline now chains traces, judges, datasets, and offline evaluations. The “Campaigns” feature and the chat_completion_events.search, campaigns.*, and import_from_explorer methods from the previous version no longer exist. Sources: Explore traces · Offline evaluations.

The end-to-end observability workflow

You have now mastered each component of Mistral’s observability stack. This final lesson assembles the pieces into a complete pipeline that you can reproduce and adapt to your production context.

Python implementation

The script below automates steps 1, 3, and 4; the judge for step 2 is created once, as in Lesson 13. Two clients coexist: the mistralai SDK client for traces and datasets, and the mistralai-evaluations client for evaluation.

import asyncio
import os
from datetime import datetime, timedelta, timezone

from mistralai.client import Mistral
from mistralai.evaluations import Dataset, Evaluation, Evaluator, Goal, Project, System
from mistralai.evaluations import Mistral as MistralEvaluations

key = os.environ["MISTRAL_API_KEY"]
api = Mistral(api_key=key)
evaluations = MistralEvaluations(api_key=key)

# 1. Identify failed traces from the week
now = datetime.now(timezone.utc)
response = api.beta.observability.traces.search(
    from_=now - timedelta(days=7),
    to=now,
    search_expression="service_name = 'agent-support' AND status_code = 'Error'",
)
for trace in response.traces.results or []:
    print(trace.trace_id, trace.root_span_name)

# 3. Add manually reviewed and corrected cases to the dataset
corrected_cases = [
    {"ticket": "Payment refused even though my card is valid.", "category": "billing"},
    {"ticket": "Unable to add a colleague to my workspace.", "category": "access"},
]
dataset = api.beta.observability.datasets.create(
    name="Support incidents — week 44",
    description="Cases drawn from failed traces, expected output corrected",
)
for case in corrected_cases:
    api.beta.observability.datasets.create_record(dataset_id=dataset.id, payload=case)


# 4. Evaluate the candidate version (task and accuracy: see Lesson 14)
async def main():
    run = await evaluations.evaluation.run(
        project=Project(name="Ticket sorting"),
        evaluation=Evaluation(name="Week 44 non-regression"),
        system=System(name="medium-prompt-v2", params={
            "model": "mistral-medium-latest",
            "system_prompt": "Classify the ticket: access, billing, or bug. Reply with a single word.",
        }),
        dataset=Dataset(id=dataset.id),
        task=task,
        evaluators=[Evaluator(name="accuracy", scorer=accuracy, goal=Goal.gte(0.9))],
    )
    run.show(level="records")


asyncio.run(main())

Run weekly, this script grows the non-regression dataset with real incidents, and every candidate version must meet its objectives on this dataset before deployment.

Course summary

This course has guided you through four essential domains for AI production with Mistral:

Batch Inference (Lessons 1-4)

You know how to process millions of requests at reduced cost, using JSONL format, file batching, and inline batching.

Predicted Outputs (Lessons 5-7)

You know how to reduce generation latency by providing an output prediction, particularly for code and text editing.

Citations and References (Lessons 8-10)

You know how to anchor your LLM’s responses in verified sources, parse citations, and build reliable RAG pipelines.

Observability (Lessons 11-16)

You know how to trace your application with OpenTelemetry, explore production traces, automate quality evaluation with judges, build datasets, and measure each version via offline evaluations.

Recommended next steps

The best way to solidify these skills is to apply them in the order that benefits your production environment most. Start by deploying a batch pipeline on a concrete use case in your organization—a monthly classification, a catalog enrichment—because it’s the quickest gain to demonstrate. Next, integrate Predicted Outputs wherever your tools edit existing code or documents: the latency gain is measurable within days. Add citations to your RAG system as soon as users rely on its responses to make decisions. Finally, observability starts with sending traces, which is possible with a simple API key; the Explorer and offline evaluations require an Enterprise organization. It is the Enterprise tier that will transform the first three initiatives into a continuous improvement loop rather than one-off deployments.

Key takeaways

• The pipeline follows the loop: trace and explore, judge, build a dataset, evaluate, iterate
• Every production incident can become a test case, and every candidate version must meet its objectives
• Automate the loop with the SDK (traces.search, datasets.create_record, evaluation.run) and run it at regular intervals
• The four domains of this course (batch, predicted outputs, citations, observability) complement each other for robust AI production

Test your knowledge

Batch, latency, citations, observability: the four pillars of production.

1. When is Batch Inference the right choice?

Answer: For high volumes without real-time constraints: asynchronous JSONL jobs (or inline up to 10K requests) at reduced cost—classification, enrichment, mass generation.

2. What are Predicted Outputs?

Answer: An acceleration when the response is largely predictable (editing existing text, templates): you provide the prediction, and the model only “pays” for the differences—latency drops significantly.

3. What do citations with Reference Object enable?

Answer: Linking every statement in the response to its sources in a RAG: verifiable answers, reduced hallucinations—trust becomes a product property.

4. What do you do in the Explorer?

Answer: Search and inspect production traces—using an expression language on service_name, status_code, duration_ns, or span attributes—to find problematic requests, then turn them into test cases: the raw material for any improvement.

5. How do judges, datasets, and evaluations work together?

Answer: Judges evaluate automatically (LLM-as-judge), datasets accumulate test cases drawn from production, and offline evaluations measure each version against these datasets with PASS/FAIL objectives—the trace → judge → build → evaluate loop.

The course’s final pipeline chains all four together: produce economically, respond quickly, source answers, observe—mature production in a single workflow.