Evaluate Your Agents: Metrics and Benchmarks

An agent that works in development can fail in production. Systematic evaluation is the only way to guarantee quality before and after deployment. In this lesson, you will learn how to define metrics, create test suites, and automate evaluation.

Why Evaluate?

Without formal evaluation, you are flying blind. You rephrase an instruction because a user complained, but nothing tells you whether the agent improved or if you just broke three other scenarios. You hesitate between Terra and Luna for your use case—or between two reasoning levels on the same model—and without measurement, the decision rests on a gut feeling after five manual trials. You change a tool schema, the agent stops calling it in a specific case, and no one notices until ticket volume spikes. An evaluation suite turns each of these situations into a comparable number from one version to the next.

Core Metrics

Accuracy

The first question is the most direct: does the agent give the right answer? You replay a set of questions where you know the answer and verify its presence in the output. Case-insensitive substring matching is intentionally coarse—it tolerates “The Pro laptop costs €1,299 incl. tax” where strict equality would fail—but it suffices for verifiable facts and calculates in a second.

from agents import Agent, Runner

async def evaluate_accuracy(agent, test_cases):
    results = []
    for case in test_cases:
        result = await Runner.run(agent, case["question"])
        # Automatic or LLM-based verification
        is_correct = case["expected_answer"].lower() in result.final_output.lower()
        results.append({
            "question": case["question"],
            "answer": result.final_output,
            "expected": case["expected_answer"],
            "correct": is_correct,
        })

    rate = sum(1 for r in results if r["correct"]) / len(results)
    print(f"Accuracy: {rate:.1%} ({sum(1 for r in results if r['correct'])}/{len(results)})")
    return results

# Test suite
test_cases = [
    {"question": "What is the price of the Pro laptop?", "expected_answer": "1299"},
    {"question": "How many laptops are in stock?", "expected_answer": "45"},
    {"question": "What is the cheapest product?", "expected_answer": "tablet"},
]

Correct Tool Usage

A correct answer obtained by chance remains a problem. An agent that recites a memorized price instead of querying the catalog will pass the accuracy test today and give an outdated rate tomorrow. The second metric therefore looks at the path, not the result: does the agent call the right tools with the right arguments? You read the calls in raw_responses, filtering for function_call outputs, and the comparison uses sets to remain indifferent to order.

async def evaluate_tools(agent, test_cases):
    results = []
    for case in test_cases:
        result = await Runner.run(agent, case["question"])

        # Extract called tools from raw responses
        called_tools = []
        for response in result.raw_responses:
            for output in getattr(response, "output", []):
                if hasattr(output, "type") and output.type == "function_call":
                    called_tools.append(output.name)

        tools_correct = set(case["expected_tools"]) == set(called_tools)
        results.append({
            "question": case["question"],
            "called_tools": called_tools,
            "expected_tools": case["expected_tools"],
            "correct": tools_correct,
        })

    rate = sum(1 for r in results if r["correct"]) / len(results)
    print(f"Correct tools: {rate:.1%}")
    return results

tool_cases = [
    {
        "question": "What is the price of the laptop?",
        "expected_tools": ["search_product"],
    },
    {
        "question": "Price of the laptop with 10% discount",
        "expected_tools": ["search_product", "calculate_discount"],
    },
]

The second test case is the most instructive: the question requires chaining two tools, and this is exactly the kind of composition that a change in description can silently break.

Latency and Cost

Finally, there is the operational dimension. An average says nothing about the real experience: it is the tail of the distribution that drives users away, when one request in a hundred takes twenty seconds. By repeating each message several times, you obtain a usable distribution and read the median as the typical behavior, and the P99 as your common worst-case scenario.

import time

async def evaluate_performance(agent, messages, iterations=10):
    latencies = []
    for message in messages:
        for _ in range(iterations):
            start = time.perf_counter()
            result = await Runner.run(agent, message)
            duration = time.perf_counter() - start
            latencies.append(duration)

    import statistics
    print(f"Average latency: {statistics.mean(latencies):.2f}s")
    print(f"P50 latency: {statistics.median(latencies):.2f}s")
    print(f"P99 latency: {sorted(latencies)[int(len(latencies) * 0.99)]:.2f}s")

LLM Evaluation (LLM-as-Judge)

These three metrics leave out anything without a single correct answer: an explanation of a return policy, a sales rephrase, an apology message. For these open-ended responses, use a model to evaluate quality. The judge receives the question and the answer, applies criteria you have written—accuracy, relevance, professionalism—and returns a structured score thanks to a Pydantic output_type. The format constraint is what makes the process usable: you get an aggregatable integer, not a comment to reread.

from pydantic import BaseModel

class Evaluation(BaseModel):
    score: int  # 1-5
    reason: str
    is_correct: bool
    is_relevant: bool
    is_professional: bool

evaluator_agent = Agent(
    name="Evaluator",
    instructions="""Evaluate the quality of the agent's response.
    Criteria:
    - Accuracy: Is the response factual and correct?
    - Relevance: Does the response answer the question asked?
    - Professionalism: Is the tone appropriate?
    Score from 1 (very poor) to 5 (excellent).""",
    model="gpt-5.6-sol",
    output_type=Evaluation,
)

async def evaluate_quality(question, agent_response):
    prompt = f"""Question asked: {question}
    Agent response: {agent_response}
    Evaluate this response."""

    result = await Runner.run(evaluator_agent, prompt)
    return result.final_output

The reason field deserves your attention during reviews: this is where the judge explains why it gave a 3 out of 5, and it is often the formulation of a flaw you had not managed to name.

Structured Test Suites

An evaluation that is not reproducible is useless. Organize your test cases in a stable format where each entry carries an ID, a category, the question, the expected answer, the expected tools, and qualitative criteria. The category does all the work here: an overall score of 4.2 out of 5 is misleadingly reassuring, whereas a detailed score reveals that the “stock” category collapses to 2.8 while “price” pulls the average up.

import json

# evaluation_dataset.json
DATASET = [
    {
        "id": "test_001",
        "category": "price",
        "question": "What is the price of the Pro laptop?",
        "expected_answer": "€1,299",
        "expected_tools": ["search_product"],
        "criteria": ["contains_price", "professional_tone"],
    },
    {
        "id": "test_002",
        "category": "stock",
        "question": "Is the Tablet Air in stock?",
        "expected_answer": "yes",
        "expected_tools": ["search_product"],
        "criteria": ["clear_answer", "stock_information"],
    },
]

async def run_complete_evaluation(agent):
    results = []
    for case in DATASET:
        result = await Runner.run(agent, case["question"])
        eval_result = await evaluate_quality(case["question"], result.final_output)
        results.append({
            **case,
            "agent_response": result.final_output,
            "score": eval_result.score,
            "evaluation": eval_result.reason,
        })

    # Report
    average_score = sum(r["score"] for r in results) / len(results)
    print(f"\nAverage score: {average_score:.1f}/5")
    for cat in set(r["category"] for r in results):
        cat_scores = [r["score"] for r in results if r["category"] == cat]
        print(f"  {cat}: {sum(cat_scores)/len(cat_scores):.1f}/5")

    return results

Run this evaluation in your continuous integration pipeline, just like your unit tests, and set a threshold below which delivery is blocked. From then on, every change—new model, tweaked instructions, added tool—compares to the previous one on the same cases, and the discussion stops revolving around impressions.

Key Takeaways

Evaluate accuracy, tool usage, latency, and cost
LLM evaluation (LLM-as-Judge) handles open-ended responses
Create structured and reproducible test suites
Automate evaluation in your CI/CD to detect regressions
Systematically compare changes (model, instructions, tools)