Metrics and Benchmarks

Lesson updated November 3, 2026 based on OpenAI documentation: temperature and top_p are only accepted with a reasoning effort of none, whereas GPT-5.6 models reason by default (medium). Calls that set these parameters must therefore specify reasoning={"effort": "none"} (API Responses) or reasoning_effort="none" (Chat Completions). Sources: Using GPT-6 · GPT-5.6 Terra.

Measuring Quality with Metrics

Beyond LLM-as-Judge scores, quantitative metrics exist to evaluate different aspects of LLM outputs. Their value isn’t that they are more accurate—a well-constructed LLM judge sees things no formula ever will—but that they are stable and cheap. Cosine similarity returns the exact same number on every run, costs almost nothing, and can therefore run on thousands of cases with every commit. You then reserve the LLM judge for dimensions that truly require it.

We will review the essential metrics before assembling them into reproducible benchmarks.

Comparing Meaning Rather Than Words

Comparing two texts character by character becomes unusable as soon as phrasing is free. Semantic similarity bypasses this problem: each text is converted into a vector by an embedding model, and we measure the angle between the two vectors. Two sentences that say the same thing with different words point in the same direction.

import openai
import numpy as np

client = openai.OpenAI()

def cosine_similarity(vec_a: list[float], vec_b: list[float]) -> float:
    """Calculates the cosine similarity between two vectors."""
    a = np.array(vec_a)
    b = np.array(vec_b)
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

def semantic_similarity(text_a: str, text_b: str) -> float:
    """Measures the semantic similarity between two texts."""
    response = client.embeddings.create(
        model="text-embedding-3-large",
        input=[text_a, text_b],
    )

    vec_a = response.data[0].embedding
    vec_b = response.data[1].embedding

    return cosine_similarity(vec_a, vec_b)

# Example
score = semantic_similarity(
    "The cat is on the rug.",
    "A feline is resting on the carpet.",
)
print(f"Similarity: {score:.3f}")  # ~0.85+

The example is instructive: two sentences with no words in common score above 0.85. This is the method’s strength, and also its trap. Semantic similarity ignores negation and nuances of degree—“the payment was accepted” and “the payment was refused” look very similar vector-wise. Therefore, never use it alone to validate a response where factual accuracy matters.

When the question is “does the answer actually address the question asked,” no formula replaces judgment. The relevance score below breaks this judgment down into three distinct questions: Is the answer direct? Does it stay focused? Is it complete? This breakdown is better than a global score because it tells you which direction to correct the prompt—a verbose model and an incomplete model call for opposite corrections.

def relevance_score(
    question: str,
    answer: str,
    context: str | None = None,
) -> dict:
    """Evaluates whether the answer is relevant to the question."""
    prompt = (
        "Evaluate the relevance of this answer.\n\n"
        f"Question: {question}\n"
    )
    if context:
        prompt += f"Context provided: {context[:1000]}\n"
    prompt += (
        f"Answer: {answer}\n\n"
        "Criteria:\n"
        "1. Does the answer directly address the question? (0-5)\n"
        "2. Does the answer contain unrequested information? (0-5, 5=none)\n"
        "3. Is the answer complete? (0-5)\n\n"
        "Respond in JSON with keys: direct, focus, completeness."
    )

    response = client.responses.create(
        model="gpt-5.6-terra",
        reasoning={"effort": "none"},  # temperature requires effort none
        input=prompt,
        temperature=0.0,
    )

    import json
    try:
        scores = json.loads(response.output_text)
        average = sum(scores.values()) / (len(scores) * 5)
        return {"score": average, "details": scores}
    except (json.JSONDecodeError, TypeError):
        return {"score": 0, "error": "Parsing failed"}

Note the inverted phrasing of the second criterion, where 5 means “no unrequested information.” This allows adding the three scores in the same direction, but it is counter-intuitive to read: when you add your own criteria, keep this convention or you will get averages that reward the flaw you intended to penalize.

Verifying the Model Isn’t Hallucinating

In a document-based Q&A application, the decisive metric isn’t writing quality but fidelity to the source. The control principle is simple: present the judge with the original context and the generated answer, and ask it to classify each statement as supported by the context, absent from the context, or contradictory to it. The distinction between these last two categories matters, because a statement merely absent might be accurate general knowledge, whereas a statement contradicting the source is always an error.

def detect_hallucinations(
    context: str,
    answer: str,
) -> dict:
    """Detects statements not supported by the context."""
    prompt = (
        "Analyze this answer and identify any statement "
        "that is NOT supported by the provided context.\n\n"
        f"Context: {context[:3000]}\n\n"
        f"Answer: {answer}\n\n"
        "For each statement in the answer, indicate:\n"
        "- supported: the statement is in the context\n"
        "- unsupported: the statement is not in the context\n"
        "- invented: the statement contradicts the context\n\n"
        "Respond in JSON with the key statements (list) and "
        "fidelity_score (0 to 1)."
    )

    response = client.responses.create(
        model="gpt-5.6-terra",
        reasoning={"effort": "none"},  # temperature requires effort none
        input=prompt,
        temperature=0.0,
    )

    import json
    try:
        return json.loads(response.output_text)
    except json.JSONDecodeError:
        return {"fidelity_score": 0, "error": "Parsing failed"}

Truncating the context to 3,000 characters is a limit you must be aware of: if the source exceeds this size, perfectly accurate statements will be counted as unsupported simply because the passage justifying them was cut off. For long documents, split and evaluate by section rather than accepting systematic false positives.

Assembling a Reproducible Benchmark

A benchmark is not just another metric: it is a setup that freezes a dataset and a protocol so that two runs are comparable. The class below measures relevance, fidelity (when context is available), latency, and cost, then keeps each result dated and labeled by prompt version.

from dataclasses import dataclass, field
import time

@dataclass
class BenchmarkResult:
    model: str
    prompt_version: str
    scores: dict
    avg_latency: float
    total_cost: float
    date: str = field(default_factory=lambda: time.strftime("%Y-%m-%d"))

class Benchmark:
    """Reusable benchmark for comparing models and prompts."""

    def __init__(self, name: str, dataset: list[dict]):
        self.name = name
        self.dataset = dataset
        self.results: list[BenchmarkResult] = []

    def run(
        self,
        model: str,
        system_prompt: str,
        prompt_version: str,
    ) -> BenchmarkResult:
        """Runs the benchmark with a given model and prompt."""
        relevance_scores = []
        hallucination_scores = []
        latencies = []
        cost = 0

        for case in self.dataset:
            start = time.perf_counter()
            response = client.responses.create(
                model=model,
                instructions=system_prompt,
                input=case["input"],
            )
            latency = time.perf_counter() - start
            latencies.append(latency)

            # Evaluate
            relevance = relevance_score(
                case["input"], response.output_text
            )
            relevance_scores.append(relevance["score"])

            if case.get("context"):
                halluc = detect_hallucinations(
                    case["context"], response.output_text
                )
                hallucination_scores.append(
                    halluc.get("fidelity_score", 0)
                )

        result = BenchmarkResult(
            model=model,
            prompt_version=prompt_version,
            scores={
                "relevance": sum(relevance_scores) / len(relevance_scores),
                "fidelity": (
                    sum(hallucination_scores) / len(hallucination_scores)
                    if hallucination_scores else None
                ),
            },
            avg_latency=sum(latencies) / len(latencies),
            total_cost=cost,
        )

        self.results.append(result)
        return result

    def compare(self) -> str:
        """Compares all benchmark results."""
        lines = [f"Benchmark: {self.name}\n"]
        for r in self.results:
            lines.append(
                f"  {r.model} (prompt {r.prompt_version}) | "
                f"Relevance: {r.scores['relevance']:.2f} | "
                f"Latency: {r.avg_latency:.2f}s"
            )
        return "\n".join(lines)

The output of compare deliberately places relevance and latency on the same line, because that is how the decision is actually made. A model that gains two hundredths of a point in relevance while doubling response time is not a better choice for an interactive application; it might be for overnight processing. The table doesn’t decide for you; it puts the trade-off right in front of your eyes.

Finally, total_cost remains zero in this implementation. Populate it from the usage fields returned by the API: a model comparison that omits cost leaves out the criterion that will settle the discussion with your finance team.

Key Takeaways

• Semantic similarity measures meaning but remains blind to negation
• Break down relevance into distinct criteria rather than a global score
• Detect hallucinations by confronting the answer with the source context
• Create reproducible benchmarks that measure quality, latency, and cost together
• Combine automatic metrics and LLM-as-Judge for a complete evaluation