Evaluate a RAG Pipeline

Objectives

• Evaluate retrieval and generation separately
• Implement automated RAG quality metrics
• Build a complete evaluation framework

The Two Axes of Evaluation

A RAG pipeline fails in two very different ways, and confusing them will cost you weeks. Either the retrieval step failed to find the right documents—in which case the answer is wrong for reasons unrelated to the generation model, and the metrics from Lesson 10 will diagnose this—or the right documents were present, but the generation step went off the rails: the model invented a detail or answered a different question than the one asked.

When a user reports “the answer is wrong,” they won’t tell you which of the two causes is at play. Hence the rule: measure both components independently with distinct metrics, and only tweak one after you’ve identified which one is underperforming.

Generation Metrics

Faithfulness answers the question that matters most in production: is the answer entirely supported by the provided documents, or did the model add its own spin? Instead of asking for a global score—which judges tend to award too generously—have the judge break down the answer claim by claim and classify each as supported, unsupported, or contradicted. That last category is the most valuable: a claim that contradicts the source signals a far more serious error than a simple omission.

from openai import OpenAI

client = OpenAI()

def evaluate_faithfulness(
    question: str,
    answer: str,
    contexts: list[str]
) -> dict:
    """Evaluates whether the answer is faithful to the contexts."""
    full_context = "\n\n---\n\n".join(contexts)

    response = client.chat.completions.create(
        model="gpt-5.6-terra",
        messages=[{
            "role": "user",
            "content": (
                f"Evaluate the faithfulness of this answer relative to the "
                f"source documents.\n\n"
                f"Source documents:\n{full_context}\n\n"
                f"Question: {question}\n\n"
                f"Answer: {answer}\n\n"
                f"For each claim in the answer, indicate:\n"
                f"- SUPPORTED: if confirmed by the documents\n"
                f"- UNSUPPORTED: if it does not appear in the documents\n"
                f"- CONTRADICTED: if it contradicts the documents\n\n"
                f"End with a global score from 0 to 1."
            )
        }],
        temperature=0
    )
    return {"evaluation": response.choices[0].message.content}

An answer can be perfectly faithful yet useless: it might faithfully copy a passage that deals with something else. That’s what answer relevance measures, this time comparing the answer to the question alone, without the documents. The prompt sets an explicit scale—1.0 for a complete answer, 0.7 for a partial but useful answer, 0.3 for tangential, 0.0 for off-topic—because a judge asked simply to “rate from 0 to 1” produces unstable scores from one run to the next.

def evaluate_relevance(question: str, answer: str) -> float:
    """Relevance score of the answer relative to the question."""
    response = client.chat.completions.create(
        model="gpt-5.6-terra",
        messages=[{
            "role": "user",
            "content": (
                f"On a scale of 0 to 1, how well does this answer "
                f"address the question asked?\n\n"
                f"Question: {question}\n"
                f"Answer: {answer}\n\n"
                f"Criteria:\n"
                f"- 1.0: complete and precise answer\n"
                f"- 0.7: partial but useful answer\n"
                f"- 0.3: tangentially related\n"
                f"- 0.0: off-topic\n\n"
                f"Return only the numeric score."
            )
        }],
        temperature=0,
        max_tokens=10
    )
    try:
        return float(response.choices[0].message.content.strip())
    except ValueError:
        return 0.0

Then there’s context precision, which bridges the two axes: among the five documents you injected into the prompt, how many were actually used? If the answer to a question about leave policies drew on only one of the five chunks, the other four consumed context budget and added noise. So you ask the judge about each context in isolation, with a simple yes or no, and calculate the proportion that were useful.

def evaluate_context_precision(
    question: str,
    contexts: list[str],
    answer: str
) -> dict:
    """Evaluates which contexts were useful for the answer."""
    results = []

    for i, ctx in enumerate(contexts):
        response = client.chat.completions.create(
            model="gpt-5.6-terra",
            messages=[{
                "role": "user",
                "content": (
                    f"Was this document useful for answering "
                    f"the question?\n\n"
                    f"Question: {question}\n"
                    f"Document: {ctx[:500]}\n"
                    f"Generated answer: {answer}\n\n"
                    f"Answer YES or NO."
                )
            }],
            temperature=0,
            max_tokens=5
        )
        useful = "yes" in response.choices[0].message.content.lower()
        results.append({"index": i, "useful": useful})

    num_useful = sum(1 for r in results if r["useful"])
    return {
        "details": results,
        "precision": num_useful / len(contexts) if contexts else 0
    }

Complete Evaluation Framework

These three functions are useless as long as they remain manual. The decisive step is to run them on a fixed set of questions every time you modify the pipeline: that’s what turns a gut feeling—“I think it’s better since I changed the chunking”—into a comparable number. The class below executes the pipeline on each item in the dataset, collects the scores, and returns their averages.

Look at the example dataset: two HR questions, each with an expected answer and the identifier of the documents that should be retrieved. Twenty to fifty entries of this type, written once and for all from real user questions, are enough to detect regressions.

class RAGEvaluator:
    def __init__(self, rag_pipeline):
        self.pipeline = rag_pipeline
        self.client = OpenAI()

    def evaluate_dataset(self, dataset: list[dict]) -> dict:
        """Evaluates the pipeline on a question-answer dataset.

        Each item: {question, expected_answer, relevant_docs}
        """
        results = {
            "faithfulness": [],
            "relevance": [],
            "context_precision": [],
        }

        for item in dataset:
            # Execute the pipeline
            output = self.pipeline.respond(item["question"])
            answer = output["answer"]
            contexts = [s["excerpt"] for s in output["sources"]]

            # Evaluate
            relevance = evaluate_relevance(item["question"], answer)
            results["relevance"].append(relevance)

            ctx_eval = evaluate_context_precision(
                item["question"], contexts, answer
            )
            results["context_precision"].append(ctx_eval["precision"])

        # Averages
        import numpy as np
        return {
            metric: round(float(np.mean(scores)), 4)
            for metric, scores in results.items()
        }

# Usage
dataset = [
    {
        "question": "How many leave days do I have?",
        "expected_answer": "25 days of paid leave per year",
        "relevant_docs": ["leave-policy"]
    },
    {
        "question": "Can I work remotely on Mondays?",
        "expected_answer": "Yes, remote work is allowed except on Tuesdays and Thursdays",
        "relevant_docs": ["remote-work-policy"]
    },
]

evaluator = RAGEvaluator(rag_pipeline)
metrics = evaluator.evaluate_dataset(dataset)
print("Results:")
for k, v in metrics.items():
    print(f"  {k}: {v}")

Metrics Dashboard

Here are the targets to aim for in a production system, along with what each number actually means.

Read this table from top to bottom as a diagnostic tree. A recall below 0.90 makes any analysis of the following lines pointless: fix the chunking or the index first. Conversely, excellent recall paired with poor faithfulness points to the generation prompt, not the search.

Summary

• Evaluate retrieval (recall, precision) and generation (faithfulness, relevance) separately
• Faithfulness detects LLM hallucinations
• Relevance verifies that the answer is useful
• Build a test dataset and measure before every change