Evaluate search quality

Objectives

• Understand standard search evaluation metrics
• Build an evaluation dataset
• Measure and improve your engine’s quality

Why evaluate?

Previous lessons introduced a series of choices: embedding model, number of dimensions, chunk size, and hybrid weighting. Each choice can improve or degrade your system. Subjective impressions from manually testing three queries aren’t enough to decide—especially since a change often improves one category of queries while degrading another. Without metrics, you’re flying blind.

Key metrics

Recall@k answers this question: among the relevant documents, how many appear in the top k results? This is the gold standard metric for RAG because a document missing from the context can never be cited by the generation model, no matter how good it is.

def recall_at_k(pertinents: set[str], resultats: list[str], k: int) -> float:
    """Proportion of relevant documents found in the top-k."""
    top_k = set(resultats[:k])
    trouves = pertinents & top_k
    return len(trouves) / len(pertinents) if pertinents else 0.0

Precision@k looks at the inverse: among the k results returned, how many are actually relevant? This matters most when the context sent to the LLM is limited, since every off-topic document takes up space and dilutes the model’s attention.

def precision_at_k(pertinents: set[str], resultats: list[str], k: int) -> float:
    """Proportion of relevant results in the top-k."""
    top_k = resultats[:k]
    trouves = sum(1 for r in top_k if r in pertinents)
    return trouves / k if k > 0 else 0.0

MRR (Mean Reciprocal Rank) measures the rank of the first relevant result. A document in first place scores 1, in second place 0.5, and in fifth place 0.2. Prioritize this metric for search bars intended for humans, who rarely look beyond the first three lines.

def mrr(pertinents: set[str], resultats: list[str]) -> float:
    """Reciprocal rank of the first relevant result."""
    for i, r in enumerate(resultats):
        if r in pertinents:
            return 1.0 / (i + 1)
    return 0.0

NDCG@k (Normalized Discounted Cumulative Gain) goes further by abandoning binary relevance: it accepts degrees of relevance, allowing you to distinguish between a document that answers the question exactly and one that only touches on it. The logarithm in the denominator applies a progressive discount based on rank, and normalization against the ideal ranking scales everything between 0 and 1.

import numpy as np

def ndcg_at_k(relevances: list[float], k: int) -> float:
    """NDCG with graded relevance scores."""
    dcg = sum(
        rel / np.log2(i + 2) for i, rel in enumerate(relevances[:k])
    )
    ideal = sorted(relevances, reverse=True)[:k]
    idcg = sum(
        rel / np.log2(i + 2) for i, rel in enumerate(ideal)
    )
    return dcg / idcg if idcg > 0 else 0.0

Create an evaluation dataset

None of these formulas are useful without ground truth. An evaluation dataset associates each query with the list of documents that should appear, and optionally those that definitely should not—the latter are useful for monitoring known false positives.

import json

# Dataset structure
dataset_eval = [
    {
        "requête": "comment installer Python sur Windows",
        "pertinents": ["doc_install_python", "doc_setup_env"],
        "non_pertinents": ["doc_cuisine"]
    },
    {
        "requête": "configurer nginx comme reverse proxy",
        "pertinents": ["doc_nginx_proxy", "doc_nginx_config"],
        "non_pertinents": ["doc_python_web"]
    },
]

# Save
with open("eval_dataset.json", "w") as f:
    json.dump(dataset_eval, f, ensure_ascii=False, indent=2)

Thirty well-chosen pairs covering the different types of questions your engine actually receives are worth more than three hundred mass-produced ones. Prioritize pulling from your search logs: these are your users’ actual phrasings, not the ones you imagine.

Generate test queries with an LLM

On a large corpus, manual writing becomes impractical. So we flip the problem: instead of looking for documents that answer a question, we ask a model what questions a given document can address. The document is by construction relevant to these questions, and you get pairs in minutes.

from openai import OpenAI

client = OpenAI()

def generer_requetes_test(document: str, n: int = 3) -> list[str]:
    """Generate n queries that this document should answer."""
    response = client.chat.completions.create(
        model="gpt-5.6-terra",
        messages=[{
            "role": "user",
            "content": (
                f"Voici un document :\n\n{document}\n\n"
                f"Génère {n} questions auxquelles ce document répond. "
                f"Retourne uniquement les questions, une par ligne."
            )
        }]
    )
    return response.choices[0].message.content.strip().split("\n")

This method has a blind spot you need to know: generated questions often reuse the document’s exact vocabulary, making them easier than real user queries and artificially inflating your scores. Use them to compare two configurations against each other, never to declare an absolute quality level.

Complete evaluation pipeline

The evaluator runs each query from the dataset on the engine, calculates all metrics for several values of k, and then averages the whole set.

class EvaluateurRecherche:
    def __init__(self, moteur_recherche):
        self.moteur = moteur_recherche

    def evaluer(
        self, dataset: list[dict], k_values: list[int] = [1, 3, 5, 10]
    ) -> dict:
        """Evaluate the engine on a complete dataset."""
        resultats = {f"recall@{k}": [] for k in k_values}
        resultats.update({f"precision@{k}": [] for k in k_values})
        resultats["mrr"] = []

        for item in dataset:
            # Run search
            res = self.moteur.rechercher(item["requête"], k=max(k_values))
            ids_resultats = [r["id"] for r in res]
            pertinents = set(item["pertinents"])

            # Calculate metrics
            resultats["mrr"].append(mrr(pertinents, ids_resultats))
            for k in k_values:
                resultats[f"recall@{k}"].append(
                    recall_at_k(pertinents, ids_resultats, k)
                )
                resultats[f"precision@{k}"].append(
                    precision_at_k(pertinents, ids_resultats, k)
                )

        # Average
        return {
            metrique: round(np.mean(scores), 4)
            for metrique, scores in resultats.items()
        }

# Usage
evaluateur = EvaluateurRecherche(moteur)
metriques = evaluateur.evaluer(dataset_eval)

print("Résultats d'évaluation :")
for metrique, score in metriques.items():
    print(f"  {metrique}: {score}")

Tracking multiple values of k is instructive in itself: a high recall@10 paired with a poor recall@1 signals that the right documents are being found but poorly ranked—a reranking problem, not an indexing problem.

Compare two configurations

The following function pits two engines against the same dataset and displays the metric gap per metric. Repeat this step before every production deployment, even for a change that seems trivial.

def comparer_configs(moteur_a, moteur_b, dataset, noms=("A", "B")):
    """Compare two search engines."""
    eval_a = EvaluateurRecherche(moteur_a).evaluer(dataset)
    eval_b = EvaluateurRecherche(moteur_b).evaluer(dataset)

    print(f"{Métrique:<20} {noms[0]:<10} {noms[1]:<10} {Delta:<10}")
    print("-" * 50)
    for metrique in eval_a:
        delta = eval_b[metrique] - eval_a[metrique]
        signe = "+" if delta > 0 else ""
        print(f"{metrique:<20} {eval_a[metrique]:<10.4f} "
              f"{eval_b[metrique]:<10.4f} {signe}{delta:<10.4f}")

Read deltas cautiously on a small dataset: a 0.02 gap over thirty queries is likely noise, whereas a 0.10 drop in recall@5 is a clear signal not to deploy.

Summary

• Recall@k, Precision@k, MRR, and NDCG are the standard metrics
• Create an evaluation dataset with (query, relevant documents) pairs
• Automate query generation with an LLM
• Systematically compare configurations before deploying