Trace grading: scoring executions

Tracing records what your agent does. That’s essential, but it’s not enough: no one reads ten thousand traces a day, and problems hide precisely in the ones you never open. Trace grading goes further: it automatically scores every execution to spot issues, measure quality over time, and feed a continuous improvement loop.

The concept of trace grading

Every trace your agent produces contains the full execution: the user message, tool calls, intermediate responses, and the final answer. Trace grading analyzes these traces and assigns scores based on your criteria. You move from a log you consult after the fact to a metric you monitor continuously, and the question “How is the agent performing right now?” stops being a guess.

Automatic grading with rules

The simplest level uses programmatic rules to score each trace. Each rule starts from a perfect score of 1.0 and deducts points based on what it observes. An execution that takes longer than ten seconds loses 0.2 points, not because the result is wrong but because the user likely gave up. More than fifteen turns costs 0.3 points: that’s the signature of an agent spinning in circles between two tools. Each tool error deducts 0.1 points. Triggered guardrails are flagged without penalty—they did their job—but a response shorter than ten characters alone costs half a point, because a silent agent is a useless agent.

from dataclasses import dataclass

@dataclass
class GradeTrace:
    trace_id: str
    score: float  # 0.0 to 1.0
    details: dict
    alerts: list[str]

def grade_trace(trace_data: dict) -> GradeTrace:
    """Scores a trace based on business rules."""
    score = 1.0
    alerts = []
    details = {}

    # Rule 1: Latency
    duration = trace_data.get("duration_ms", 0)
    if duration > 10000:
        score -= 0.2
        alerts.append(f"High latency: {duration}ms")
    details["latency_ms"] = duration

    # Rule 2: Number of turns
    num_turns = trace_data.get("num_turns", 0)
    if num_turns > 15:
        score -= 0.3
        alerts.append(f"Too many turns: {num_turns}")
    details["num_turns"] = num_turns

    # Rule 3: Tool errors
    tool_errors = trace_data.get("tool_errors", 0)
    if tool_errors > 0:
        score -= 0.1 * tool_errors
        alerts.append(f"{tool_errors} tool error(s)")
    details["tool_errors"] = tool_errors

    # Rule 4: Triggered guardrails
    guardrails = trace_data.get("guardrails_triggered", 0)
    if guardrails > 0:
        alerts.append(f"{guardrails} guardrail(s) triggered")
    details["guardrails"] = guardrails

    # Rule 5: Empty response
    response = trace_data.get("final_output", "")
    if not response or len(response) < 10:
        score -= 0.5
        alerts.append("Empty or too short response")
    details["response_length"] = len(response)

    return GradeTrace(
        trace_id=trace_data["trace_id"],
        score=max(0.0, score),
        details=details,
        alerts=alerts,
    )

The details dictionary always accompanies the score: without it, you’d know an execution scored 0.5 but not why. These thresholds are yours, not universal constants—a research agent tolerates fifteen turns where a checkout assistant must reply in three.

LLM-based grading

Rules don’t see content. A fast, short, polite response that misses the point entirely will score 1.0. For finer evaluation, use a model to score traces on substance: relevance, accuracy, and tone, each from 1 to 5, along with identified issues and improvement suggestions. Again, structured output via Pydantic makes scores aggregable instead of producing free-form commentary.

from pydantic import BaseModel
from agents import Agent, Runner

class GradeLLM(BaseModel):
    relevance_score: int  # 1-5
    accuracy_score: int  # 1-5
    tone_score: int  # 1-5
    issues: list[str]
    suggestions: list[str]

grader_agent = Agent(
    name="Trace Grader",
    instructions="""You score executions of a sales agent.

    Analyze the full trace (question, tools called, response) and score:
    - Relevance (1-5): Does the response answer the question?
    - Accuracy (1-5): Is the information correct?
    - Tone (1-5): Is the tone professional and appropriate?

    Identify issues and propose improvements.""",
    model="gpt-5.6-sol",
    output_type=GradeLLM,
)

async def grade_trace_llm(trace: dict) -> GradeLLM:
    prompt = f"""Execution trace to score:

    User question: {trace['user_message']}
    Tools called: {trace['tools_called']}
    Final response: {trace['final_output']}
    Duration: {trace['duration_ms']}ms
    Number of turns: {trace['num_turns']}"""

    result = await Runner.run(grader_agent, prompt)
    return result.final_output

The prompt also passes the tools called and duration: the judge can thus fault the agent for answering without consulting the catalog, which reading the response alone wouldn’t allow.

Grading pipeline in production

Now combine both approaches without blowing your budget. Scoring every trace with an LLM at production volume multiplies your inference bill; using only rules leaves you blind to substance. The pipeline below applies a filter: all traces go through programmatic grading, which is fast and free, and only those falling below 0.7 go to the judge. The judge’s score, the sum of three 1–5 ratings, is scaled to 0–1 by dividing by 15 to stay comparable to the first. Below 0.5, the trace joins the critical alerts list.

import asyncio
from collections import defaultdict

class GradingPipeline:
    def __init__(self):
        self.scores = defaultdict(list)
        self.critical_alerts = []

    async def grade_batch(self, traces: list[dict]):
        """Scores a batch of traces."""
        for trace in traces:
            # Programmatic grading (fast)
            rule_grade = grade_trace(trace)

            # LLM grading for borderline cases
            if rule_grade.score < 0.7:
                llm_grade = await grade_trace_llm(trace)
                final_score = (
                    llm_grade.relevance_score +
                    llm_grade.accuracy_score +
                    llm_grade.tone_score
                ) / 15.0  # Normalize to 0-1
            else:
                final_score = rule_grade.score

            self.scores["global"].append(final_score)

            # Alerts
            if final_score < 0.5:
                self.critical_alerts.append({
                    "trace_id": trace["trace_id"],
                    "score": final_score,
                    "alerts": rule_grade.alerts,
                })

    def report(self):
        """Generates a quality report."""
        scores = self.scores["global"]
        if not scores:
            return "No traces evaluated."

        import statistics
        return {
            "num_traces": len(scores),
            "avg_score": statistics.mean(scores),
            "median_score": statistics.median(scores),
            "min_score": min(scores),
            "pct_critical": len(self.critical_alerts) / len(scores),
            "critical_alerts": self.critical_alerts[:10],
        }

# Usage
pipeline = GradingPipeline()
await pipeline.grade_batch(traces_of_the_day)
print(pipeline.report())

The report keeps the minimum score and a sample of the ten worst traces: that’s where an investigation starts, never with the average.

Quality dashboard

A single score means nothing; its trajectory informs. By writing the daily report to a JSONL file, you build a time series that any visualization tool can read, and you can automatically compare the latest point to the previous one. A drop of more than 0.1 triggers a regression alert—and correlating it with your day’s deployments usually points to the culprit within minutes.

import json
from datetime import datetime

def log_metrics(report: dict, file: str = "agent_metrics.jsonl"):
    """Logs daily metrics."""
    entry = {
        "date": datetime.now().isoformat(),
        **report,
    }
    with open(file, "a") as f:
        f.write(json.dumps(entry) + "\n")

def detect_regression(recent_metrics: list[dict], threshold: float = 0.1):
    """Detects a significant quality drop."""
    if len(recent_metrics) < 2:
        return False
    latest = recent_metrics[-1]["avg_score"]
    previous = recent_metrics[-2]["avg_score"]
    drop = previous - latest
    if drop > threshold:
        print(f"REGRESSION DETECTED: score dropped from {previous:.2f} to {latest:.2f}")
        return True
    return False

Comparing two consecutive days remains sensitive to noise: on low volume, prefer a moving average before crying regression, or you’ll chase a statistical blip.

Key takeaways

Trace grading automatically scores every execution of your agent
Programmatic rules (latency, errors, length) are fast and reliable
LLM grading evaluates relevance, accuracy, and tone
Combine both: fast rules for filtering, LLM for borderline cases
Track scores over time to detect regressions
Critical alerts (score < 0.5) must trigger an investigation