Evaluate Your Prompts: Methods, Graders, and Migrating to Promptfoo

Lesson updated November 3, 2026 based on OpenAI documentation: temperature and top_p are only accepted with a reasoning effort of none, whereas GPT-5.6 models reason by default (medium). Calls that set these parameters must therefore specify reasoning={"effort": "none"} (Responses API) or reasoning_effort="none" (Chat Completions). Sources: Using GPT-6 · GPT-5.6 Terra.

Update November 2, 2026 — Evals Platform. OpenAI has announced the end of its Evals platform: existing evaluations have been read-only since October 31, 2026, and the dashboard along with the API will close on November 30, 2026, including graders. OpenAI has published a migration guide to Promptfoo. The evaluation principles in this lesson remain valid; only the tool is changing.

Why you must evaluate systematically

You can’t improve what you don’t measure. When you change a prompt, a model, or a parameter, how do you know quality has actually improved? The spontaneous method is to try three or four examples, find the better answers, and deploy. This fails regularly because the tested examples are precisely the ones that caused problems last week: you measure improvement where you looked for it, and you miss regressions introduced in cases that were already working.

Evaluations (evals) turn this intuition into an objective measure. Until 2026, OpenAI offered an integrated platform for this, Evals; it has been read-only since October 31, 2026, and closes on November 30. OpenAI now recommends Promptfoo, an open-source tool for evaluation and red teaming. The good news is that both share the same building blocks—test data, prompts, models, and scoring criteria. What you learn here in Python code translates directly, and the final part of the lesson shows the move to Promptfoo.

Three concepts structure the whole process. The dataset is a set of test cases with inputs and expected outputs: it’s your reference, the equivalent of a unit test suite. The grader is an evaluation criterion—exact match, LLM-as-judge, score—that decides whether a given response is good. The run is an evaluation execution on a dataset with a specific model and prompt; it’s the unit you compare from one version to the next.

Build a dataset that looks like your real world

A test case consists of an input, an expected output, and metadata. The latter aren’t decorative: they’re what will let you realize, after the fact, that an overall score of 87% hides a collapse in the “code” category compensated by excellent results in geography. Without category or difficulty, you only have an aggregated number, and an aggregated number never tells you where to intervene.

import json

# A dataset is a list of test cases
dataset = [
    {
        "input": "What is the capital of France?",
        "expected": "Paris",
        "metadata": {"category": "geography", "difficulty": "easy"},
    },
    {
        "input": "Explain Bayes' theorem in one sentence.",
        "expected": "Bayes' theorem calculates the probability "
                    "of an event based on known conditional probabilities.",
        "metadata": {"category": "mathematics", "difficulty": "medium"},
    },
    {
        "input": "Write a Python function that reverses a list.",
        "expected": "def reverse_list(lst): return lst[::-1]",
        "metadata": {"category": "code", "difficulty": "easy"},
    },
]

# Save as JSONL
with open("eval_dataset.jsonl", "w") as f:
    for case in dataset:
        f.write(json.dumps(case, ensure_ascii=False) + "\n")

A dataset invented at the office evaluates your imagination, not your application. Your users phrase their requests differently than you do: typos, half-asked questions, implicit context. The best source of test cases is therefore your own production data, provided you have a human validate the reference answers—otherwise you freeze current errors as the norm.

import openai
import random

client = openai.OpenAI()

def create_dataset_from_logs(
    production_logs: list[dict],
    sample_size: int = 100,
) -> list[dict]:
    """Creates an evaluation dataset from production logs."""
    # Sample
    sample = random.sample(
        production_logs,
        min(sample_size, len(production_logs)),
    )

    dataset = []
    for log in sample:
        dataset.append({
            "input": log["prompt"],
            "expected": log["validated_response"],  # Response validated by a human
            "metadata": {
                "source": "production",
                "date": log["date"],
                "original_model": log["model"],
            },
        })

    return dataset

Choose the right grader

The exact match grader is the strictest and fastest. It only suits tasks where the output is constrained: a three-category classification, extracting an invoice number, a yes-or-no answer. Used on free text, it would give zero to a perfect answer simply phrased differently than the reference.

def exact_grader(response: str, expected: str) -> dict:
    """Evaluates if the response matches exactly."""
    score = 1.0 if response.strip().lower() == expected.strip().lower() else 0.0
    return {"score": score, "type": "exact"}

The inclusion grader relaxes the constraint: it checks for the presence of expected keywords and returns a partial score. This is the right tool for responses that must cover mandatory points without the wording mattering—a support response that must mention the withdrawal period and return address, for example. Its weakness is well known: a text containing the right words in an absurd order would get a perfect score.

def inclusion_grader(response: str, keywords: list[str]) -> dict:
    """Checks that the response contains the expected keywords."""
    response_lower = response.lower()
    found = sum(1 for word in keywords if word.lower() in response_lower)
    score = found / len(keywords) if keywords else 0.0
    return {
        "score": score,
        "found": found,
        "total": len(keywords),
        "type": "inclusion",
    }

Then there is the most powerful: another LLM evaluates the quality of the response. The LLM-as-Judge grader appreciates what no mechanical rule can—clarity, completeness, tone accuracy. Two implementation details make it practical. First, temperature=0.0: without this, the same judge would assign different scores to the same response from one run to the next, and your benchmark would lose all comparative value. Second, the fallback for json.JSONDecodeError: a judge that comments instead of returning JSON shouldn’t crash the entire campaign.

def llm_judge_grader(
    question: str,
    response: str,
    expected: str,
    criteria: str = "accuracy, completeness, clarity",
) -> dict:
    """Uses an LLM to evaluate response quality."""
    judge_prompt = f"""Rate the quality of this response on a scale of 1 to 5.

Question: {question}
Expected Response: {expected}
Response to Evaluate: {response}

Criteria: {criteria}

Respond ONLY with JSON:
{{"score": <1-5>, "justification": "<short explanation>"}}"""

    response_obj = client.responses.create(
        model="gpt-5.6-terra",
        reasoning={"effort": "none"},  # temperature requires effort none
        input=judge_prompt,
        temperature=0.0,
    )

    import json
    try:
        result = json.loads(response_obj.output_text)
        result["normalized_score"] = result["score"] / 5.0
        return result
    except json.JSONDecodeError:
        return {"score": 0, "justification": "Parsing error", "normalized_score": 0}

Assemble the full evaluation

Orchestration presents no conceptual difficulty: you iterate through the dataset, call the model with the system prompt to test, apply the grader, and aggregate. The interesting point is that the scoring function comes in as a parameter, which lets you replay the exact same dataset with different criteria depending on the campaign—inclusion for a quick check during development, LLM judge for validation before production.

async def run_eval(
    dataset: list[dict],
    model: str,
    system_prompt: str,
    grader_fn,
) -> dict:
    """Runs a full evaluation on a dataset."""
    results = []

    for case in dataset:
        response = client.responses.create(
            model=model,
            instructions=system_prompt,
            input=case["input"],
        )

        score = grader_fn(response.output_text, case["expected"])
        results.append({
            "input": case["input"],
            "expected": case["expected"],
            "obtained": response.output_text,
            **score,
        })

    # Calculate aggregated metrics
    scores = [r["score"] for r in results if "score" in r]
    return {
        "model": model,
        "num_cases": len(dataset),
        "average_score": sum(scores) / len(scores) if scores else 0,
        "min_score": min(scores) if scores else 0,
        "max_score": max(scores) if scores else 0,
        "results": results,
    }

The report keeps min_score and the detail of each case, and that’s deliberate. The average tells you if you’re moving forward; the worst-scored cases tell you why. Make it a habit to read the five worst responses from each run before commenting on the average: that’s where, and rarely elsewhere, you’ll find the next fix to make.

Industrialize with Promptfoo

Writing your evals by hand in Python is educational; maintaining them over time is more comfortable with a dedicated tool. OpenAI’s migration guide describes Promptfoo’s logic: the evaluation lives in a configuration file, promptfooconfig.yaml, which brings together prompts, model(s), test cases, and criteria—called assertions here. It runs from the command line, locally or in your continuous integration, and lives alongside the application code.

description: Support ticket classification
prompts:
  - "Classify this ticket into a single category (billing, technical, account): {{ticket}}"
providers:
  - openai:gpt-6.1-sol
tests:
  - vars:
      ticket: "My card was charged twice this month."
    assert:
      - type: icontains
        value: billing
  - vars:
      ticket: "The app crashes when I open a PDF."
    assert:
      - type: icontains
        value: technical

promptfoo eval -c promptfooconfig.yaml --no-cache
promptfoo view
promptfoo validate config -c promptfooconfig.yaml   # in case of configuration error

The graders in this lesson reappear as assertions: exact match, keyword inclusion, and model judgment. OpenAI warns on the last point: a recreated grader, especially an LLM judge, must be revalidated before being used to decide on a regression, because scores won’t necessarily be identical from one tool to another.

Key takeaways

• Evals turn intuition into an objective measure of quality
• Create your datasets from real production cases, validated by a human
• Choose the grader based on the task: exact for constrained, inclusion for mandatory points, LLM-as-Judge for nuance
• Set temperature=0.0 on the judge to make scores reproducible
• Run evals before every prompt or model change
• OpenAI’s Evals platform closes on November 30, 2026: Promptfoo, recommended by OpenAI, takes over datasets, prompts, models, and criteria in a configuration file