Introduction to Observability: Traces, Judges, and Evaluations
On this page
Rewritten on November 2, 2026 based on Mistral’s Observability documentation and the
mistralaiSDK 3.7.0: observability now relies on OpenTelemetry traces explored in Studio, LLM judges, and offline evaluations (mistralai-evaluations). The “Campaigns” andchat_completion_eventsAPI from previous lessons no longer exist. Sources: Distributed tracing · Offline evaluations.
Why Observability Is Essential
Your standard dashboards—latency, error rates, availability—can stay green while an assistant gives irrelevant answers, invents procedures, or shifts tone after a prompt update. An LLM in production rarely fails with an exception; it fails by serving a wrong answer with confidence. LLM observability makes these failures visible, measurable, and fixable.
The Three Pillars of Observability
1. Visibility—See What’s Happening
Every request from your application becomes a trace: a tree of spans representing each step of execution—model call, tool call, workflow step—along with its input, output, duration, token count, and status. These traces follow the OpenTelemetry standard, and Studio’s Explorer lets you search and inspect them.
2. Quality Signals—Measure Automatically
Manually reviewing thousands of conversations is impossible. Judges are LLM-based evaluators that you configure once—with instructions, a model, and an output format—and then apply to conversations or dataset recordings to get a label or score.
3. Iteration Loops—Improve Continuously
Datasets gather test cases, often pulled from real traces, and offline evaluations measure a version of your application—model, prompt, parameters—against them before deployment. You compare versions, keep the best one, and repeat.
The Four Components
| Component | Role | Analogy |
|---|---|---|
| Traces and Explorer | Collect, search, and inspect activity | An APM for AI |
| Judges | Evaluate quality automatically | A tireless reviewer |
| Datasets | Store test cases | A regression test suite |
| Evaluations | Measure a version against a dataset | CI for AI quality |
Prerequisites and Status
Keep three points in mind before you start:
- Status: Mistral’s observability is in Private Preview, and its SDK integration changes quickly. Mistral recommends using the latest SDK version.
- Sending traces requires only a valid API key. Tracing is disabled by default and must be enabled per source (covered in the next lesson).
- Reading traces in Studio requires an Enterprise organization and an appropriate role (Org Admin, Workspace Admin, or Observability Viewer). Offline evaluations and adding spans to a dataset are also limited to Enterprise organizations. Traces are retained for 30 days.
If your organization doesn’t have access yet, this section is still worth your time: the concepts—tracing, judging, building test sets, and evaluating before deployment—apply to any OpenTelemetry tooling.
The Typical Workflow
The components form a continuous loop:
- Trace and explore—Instrument your application, then use the Explorer to spot problematic traces (errors, slowness, questionable responses)
- Judge—Create a judge that formalizes the relevant quality criterion
- Build a dataset—Turn real spans into test cases
- Evaluate—Measure the current version, then each variant, against this dataset
Each cycle brings you closer to a more reliable, better-calibrated system.
Key Takeaways
- LLM observability exposes failures that standard monitoring misses: bad answers
- OpenTelemetry traces (and their Explorer), judges, datasets, and offline evaluations form the improvement loop
- Mistral’s observability is in Private Preview; sending traces requires only an API key, while reading traces and running offline evaluations require an Enterprise organization
- Traces are retained for 30 days