Create Custom Evaluations

Lesson updated November 3, 2026 based on OpenAI documentation: temperature and top_p are only accepted with a reasoning effort of none, whereas GPT-5.6 models reason by default (medium). Calls that set these parameters must therefore specify reasoning={"effort": "none"} (API Responses) or reasoning_effort="none" (Chat Completions). Sources: Using GPT-6 · GPT-5.6 Terra.

November 2, 2026 update — Evals platform. OpenAI has announced the end of its Evals platform: existing evaluations have been read-only since October 31, 2026, and both the dashboard and the API, including graders, will close on November 30, 2026. OpenAI has published a migration guide to Promptfoo. The evaluation principles in this lesson remain valid; only the tool is changing.

Tailor evaluations to your domain

Generic graders aren’t enough to assess quality in your specific domain. A medical chatbot, a legal assistant, and a marketing writing tool each have different quality criteria, and these criteria aren’t just more numerous—they’re prioritized differently. A tone mismatch in a newsletter is an annoyance; the same approximation in a dosage instruction is a serious error. A generic grader treats both the same way, which amounts to not evaluating anything at all.

That’s why you need custom evaluations built on an explicit rubric that translates your business priorities into weights. That’s the rubric we’re going to build.

Translate a business requirement into a rubric

A rubric is a list of weighted criteria that sum to 1. The most useful exercise isn’t coding the structure, which takes about fifteen lines, but filling it in: deciding that customer benefits weigh 0.25 while length weighs only 0.10 forces an explicit conversation with product owners. What you end up with is a shared definition of “a good response” that no one can dispute after the fact.

from dataclasses import dataclass

@dataclass
class Critere:
    nom: str
    description: str
    poids: float  # 0 to 1, sum of weights = 1

@dataclass
class Rubrique:
    nom: str
    criteres: list[Critere]

    def valider(self) -> bool:
        total = sum(c.poids for c in self.criteres)
        return abs(total - 1.0) < 0.01

# Example: rubric for a marketing writing assistant
rubrique_marketing = Rubrique(
    nom="Qualité rédaction marketing",
    criteres=[
        Critere("ton", "Le ton est professionnel et engageant", 0.20),
        Critere("cta", "Contient un appel à l'action clair", 0.15),
        Critere("benefices", "Met en avant les bénéfices client", 0.25),
        Critere("longueur", "Respecte la longueur demandée", 0.10),
        Critere("grammaire", "Aucune faute de grammaire ou orthographe", 0.15),
        Critere("marque", "Respecte les guidelines de la marque", 0.15),
    ],
)

The valider method deserves to be called in tests, not just written. Rubrics evolve: you add a compliance criterion, remove one that’s no longer useful, and the sum drifts to 1.15 with nothing flagging the anomaly. Scores remain calculable, just wrong, and when compared across versions they’ll lead you to erroneous conclusions.

The grader then translates the rubric into a prompt, asks for a 0–5 score per criterion, and recomposes a weighted global score. Each score comes with a short comment: that’s what will let you, when reviewing bad cases, understand whether the model responded poorly or whether your criterion was poorly worded.

import openai
import json

client = openai.OpenAI()

def evaluer_multi_criteres(
    question: str,
    reponse: str,
    rubrique: Rubrique,
) -> dict:
    """Evaluates a response against a multi-criteria rubric."""

    criteres_texte = "\n".join(
        f"- {c.nom} (poids: {c.poids}): {c.description}"
        for c in rubrique.criteres
    )

    prompt = (
        "Évaluez cette réponse selon les critères suivants.\n"
        "Pour chaque critère, donnez un score de 0 à 5.\n\n"
        f"Question posée : {question}\n"
        f"Réponse à évaluer : {reponse}\n\n"
        f"Critères :\n{criteres_texte}\n\n"
        "Répondez en JSON avec un objet scores contenant pour chaque critère "
        "un score (0-5) et un commentaire court."
    )

    response = client.responses.create(
        model="gpt-5.6-terra",
        reasoning={"effort": "none"},  # temperature requires effort none
        input=prompt,
        temperature=0.0,
    )

    try:
        resultat = json.loads(response.output_text)
        scores = resultat.get("scores", {})

        # Calculate weighted score
        score_pondere = 0
        for critere in rubrique.criteres:
            if critere.nom in scores:
                score_brut = scores[critere.nom]["score"] / 5.0
                score_pondere += score_brut * critere.poids

        return {
            "score_global": score_pondere,
            "scores_details": scores,
            "rubrique": rubrique.nom,
        }
    except (json.JSONDecodeError, KeyError):
        return {"score_global": 0, "erreur": "Parsing échoué"}

Watch out for the calculation loop: a criterion missing from the judge’s response is silently ignored and contributes zero to the weighted score. An excellent response could therefore cap at 0.75 simply because the judge forgot a key. If your scores seem stuck under an unexplained ceiling, inspect scores_details before blaming the evaluated model.

When the machine judges better than a human

Having an LLM grade code is a costly shortcut: the only verdict that matters is execution. The grader below concatenates the generated code and a test, writes it to a temporary file, executes it, and counts successes. The resulting score needs no interpretation—a test either passes or it doesn’t.

import subprocess
import tempfile

def evaluer_code_python(
    code_genere: str,
    tests: list[str],
) -> dict:
    """Evaluates Python code by executing it against tests."""
    resultats_tests = []

    for i, test in enumerate(tests):
        code_complet = f"{code_genere}\n\n{test}"

        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".py", delete=False
        ) as f:
            f.write(code_complet)
            f.flush()

            try:
                resultat = subprocess.run(
                    ["python3", f.name],
                    capture_output=True,
                    text=True,
                    timeout=10,
                )
                succes = resultat.returncode == 0
                resultats_tests.append({
                    "test": i,
                    "succes": succes,
                    "sortie": resultat.stdout[:200],
                    "erreur": resultat.stderr[:200] if not succes else "",
                })
            except subprocess.TimeoutExpired:
                resultats_tests.append({
                    "test": i,
                    "succes": False,
                    "erreur": "Timeout (10s)",
                })

    nb_succes = sum(1 for r in resultats_tests if r["succes"])
    return {
        "score": nb_succes / len(tests) if tests else 0,
        "tests_passes": nb_succes,
        "tests_total": len(tests),
        "details": resultats_tests,
    }

The timeout=10 isn’t a comfort precaution: a model regularly produces infinite loops, and without this bound your evaluation campaign gets stuck on the first failing case. Also remember that you’re executing code you didn’t write—outside an isolated development environment, this function requires a sandbox.

The summary illustrates a mixed case, where part of the judgment is mechanical and part isn’t. Length is measurable: the calculated ratio penalizes a summary that’s too short and one that’s too long symmetrically, which avoids the model’s trick of copying the source text to omit nothing. Fidelity, on the other hand, isn’t measurable by a rule and goes back to the LLM judge, with a direct question about invented information.

def evaluer_resume(
    texte_original: str,
    resume: str,
    longueur_cible: int = 100,
) -> dict:
    """Evaluates summary quality."""
    mots_resume = len(resume.split())
    ratio_longueur = min(
        mots_resume / longueur_cible,
        longueur_cible / max(mots_resume, 1),
    )

    # Fidelity via LLM
    response = client.responses.create(
        model="gpt-5.6-terra",
        input=(
            f"Le résumé suivant est-il fidèle au texte original ? "
            f"Contient-il des informations inventées ?\n\n"
            f"Texte original (extrait) : {texte_original[:2000]}\n\n"
            f"Résumé : {resume}\n\n"
            f"Répondez en JSON avec les clés : fidélité (0-5), "
            f"hallucinations (oui/non), commentaire (court)."
        ),
        temperature=0.0,
    )

    try:
        fidelite = json.loads(response.output_text)
    except json.JSONDecodeError:
        fidelite = {"fidelite": 0, "hallucinations": "inconnu"}

    return {
        "score_longueur": ratio_longueur,
        "score_fidelite": fidelite.get("fidelite", 0) / 5.0,
        "hallucinations": fidelite.get("hallucinations", "inconnu"),
        "mots_resume": mots_resume,
        "longueur_cible": longueur_cible,
    }

Make evaluation repeatable

An evaluation you rerun manually eventually stops getting rerun. The pipeline encapsulates the rubric and keeps a history of campaigns, turning a one-off measurement into a time series: you see that the average score dropped from 0.81 to 0.79 after Tuesday’s prompt change, information no isolated execution could have given you.

class PipelineEval:
    """Reusable evaluation pipeline."""

    def __init__(self, nom: str, rubrique: Rubrique):
        self.nom = nom
        self.rubrique = rubrique
        self.historique: list[dict] = []

    def evaluer_batch(
        self,
        dataset: list[dict],
        modele: str,
        prompt_systeme: str,
    ) -> dict:
        """Evaluates a full batch and returns metrics."""
        resultats = []

        for cas in dataset:
            response = client.responses.create(
                model=modele,
                instructions=prompt_systeme,
                input=cas["input"],
            )

            eval_result = evaluer_multi_criteres(
                cas["input"],
                response.output_text,
                self.rubrique,
            )
            resultats.append(eval_result)

        scores = [r["score_global"] for r in resultats]
        rapport = {
            "pipeline": self.nom,
            "modele": modele,
            "nb_cas": len(dataset),
            "score_moyen": sum(scores) / len(scores),
            "score_min": min(scores),
            "score_max": max(scores),
        }

        self.historique.append(rapport)
        return rapport

One tip to get started: build your first rubric on just twenty cases, have it scored in parallel by the LLM judge and by a domain expert, then compare the two series. The discrepancies will tell you which criteria are poorly described long before they tell you anything about the model’s quality.

Key takeaways

• Create evaluation rubrics specific to your business domain, with explicit weights
• Verify that the sum of weights stays at 1 as the rubric evolves
• Combine automated evaluation (code, length) and LLM-as-Judge (quality)
• Test generated code by executing it, with a timeout and an isolated environment
• Build a reusable pipeline that keeps a history of campaigns