Automated Prompt Evaluation
On this page
Lesson updated November 3, 2026 based on OpenAI documentation:
temperatureandtop_pare only accepted with a reasoning effort ofnone, whereas GPT-5.6 models reason by default (medium). Calls that set these parameters must therefore specifyreasoning={"effort": "none"}(Responses API) orreasoning_effort="none"(Chat Completions). Sources: Using GPT-6 · GPT-5.6 Terra.
Automated Prompt Evaluation
Automated evaluation is the cornerstone of a mature prompt engineering pipeline. It lets you detect regressions, compare models, and validate changes before deployment. Without it, every prompt change is a blind gamble.
Types of Evaluation
Three families of evaluators divide the labor based on the nature of the expected output, and you will almost always combine several of them.
Exact Match Evaluation
For tasks with a single expected answer, direct comparison suffices. eval_exact_match normalizes case and whitespace before comparing: without this precaution, “Bug” and “bug” would count as an error, sending you to fix a problem that doesn’t exist.
eval_set_match handles cases where the answer is a set—entities found in a document, categories assigned to an article. It derives precision and recall, two figures that tell different stories: high precision with low recall signals an overly cautious model that extracts only what it’s sure of; the reverse betrays a model that hallucinates to fill gaps. F1 summarizes them conveniently but doesn’t replace them for diagnosis.
def eval_exact_match(prediction: str, reference: str) -> bool:
"""Exact match after normalization."""
return prediction.strip().lower() == reference.strip().lower()
def eval_set_match(prediction: set, reference: set) -> dict:
"""Set match (entities, categories)."""
tp = len(prediction & reference)
fp = len(prediction - reference)
fn = len(reference - prediction)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = (2 * precision * recall / (precision + recall)
if (precision + recall) > 0 else 0)
return {"precision": precision, "recall": recall, "f1": f1}Semantic Evaluation (Embedding)
For text responses where wording can vary, exact match becomes absurd: “Your order will be delivered tomorrow” and “Delivery is scheduled for tomorrow” say the same thing yet would score zero. Cosine similarity between embeddings measures semantic proximity rather than character matching, returning a continuous score.
You set the threshold above which a response is deemed acceptable. Calibrate it on a few dozen pairs you’ve manually classified as “equivalent” or “different,” then pick the value that best separates the two groups: a threshold chosen by gut feeling produces results as arbitrary as no measurement at all, with the added illusion of rigor.
from openai import OpenAI
import numpy as np
client = OpenAI()
def eval_semantic_similarity(prediction: str,
reference: str) -> float:
"""Cosine similarity between embeddings."""
response = client.embeddings.create(
model="text-embedding-3-large",
input=[prediction, reference]
)
emb_pred = np.array(response.data[0].embedding)
emb_ref = np.array(response.data[1].embedding)
similarity = float(
np.dot(emb_pred, emb_ref) /
(np.linalg.norm(emb_pred) * np.linalg.norm(emb_ref))
)
return similarityLLM Evaluation (LLM-as-Judge)
The model evaluates the quality of a response. This is the only viable approach when no reference exists: no one will hand-write the “correct” support response for two hundred tickets.
What makes a judge usable comes down to a few things. First, the model used: you pick a more capable model as judge, here gpt-5.6-sol, because a judge at the same level as the evaluatee tends to validate its own flaws. Second, requiring justification: motivating each score stabilizes judgment and lets you spot an evaluator using the wrong criterion—a justification discussing length when the criterion was clarity is a red flag. Finally, temperature at 0.1 ensures score stability across runs.
def eval_llm_judge(question: str, prediction: str,
criteria: list[str]) -> dict:
"""Evaluation by a judge model."""
criteria_text = "\n".join(
f"- {c}: score from 1 to 5" for c in criteria
)
response = client.responses.create(
model="gpt-5.6-sol", # More capable model as judge
reasoning={"effort": "none"}, # temperature requires effort none
instructions="You are an expert, impartial evaluator.",
input=f"""Evaluate this response according to the following criteria:
Question asked: {question}
Response to evaluate: {prediction}
Criteria:
{criteria_text}
For each criterion, give a score from 1 to 5 with justification.""",
text={
"format": {
"type": "json_schema",
"name": "evaluation",
"schema": {
"type": "object",
"properties": {
"scores": {
"type": "array",
"items": {
"type": "object",
"properties": {
"criterion": {"type": "string"},
"score": {"type": "integer"},
"justification": {"type": "string"}
},
"required": ["criterion", "score",
"justification"],
"additionalProperties": False
}
},
"overall_score": {"type": "number"},
"verdict": {"type": "string"}
},
"required": ["scores", "overall_score", "verdict"],
"additionalProperties": False
},
"strict": True
}
},
temperature=0.1
)
return json.loads(response.output_text)Complete Evaluation Pipeline
EvalPipeline assembles these building blocks: it loads a test suite, runs the prompt on each case, applies all provided evaluators, and aggregates scores. Evaluators are passed as a list of functions, all called with the same signature; adding a metric therefore requires no pipeline modification, and you can run exact match, semantic similarity, and a judge on the same execution without paying for generation three times.
The summary method also accumulates consumed tokens. This seemingly secondary line protects you from a common trap: a prompt version that gains two score points while tripling the bill isn’t necessarily the right choice, and it’s better to learn that at trade-off time than when the monthly invoice arrives.
import json
from dataclasses import dataclass
from pathlib import Path
@dataclass
class TestCase:
"""A test case for evaluation."""
input_text: str
expected_output: str = None
metadata: dict = None
class EvalPipeline:
"""Automated evaluation pipeline."""
def __init__(self, test_suite_path: str):
"""Loads the test suite from a JSON file."""
data = json.loads(Path(test_suite_path).read_text())
self.test_cases = [TestCase(**tc) for tc in data]
self.results = []
def run(self, prompt_config: dict,
evaluators: list[callable]) -> dict:
"""Runs all tests and evaluates results."""
self.results = []
for tc in self.test_cases:
# Generate response
response = client.responses.create(
model=prompt_config["model"],
reasoning={"effort": "none"}, # temperature requires effort none
instructions=prompt_config["system_prompt"],
input=tc.input_text,
temperature=prompt_config.get("temperature", 0.1)
)
prediction = response.output_text
# Evaluate with each evaluator
scores = {}
for evaluator in evaluators:
score = evaluator(
question=tc.input_text,
prediction=prediction,
reference=tc.expected_output
)
scores[evaluator.__name__] = score
self.results.append({
"input": tc.input_text,
"expected": tc.expected_output,
"prediction": prediction,
"scores": scores,
"tokens": {
"input": response.usage.input_tokens,
"output": response.usage.output_tokens
}
})
return self.summary()
def summary(self) -> dict:
"""Summarizes evaluation results."""
if not self.results:
return {"error": "No results"}
# Aggregate scores by evaluator
all_scores = {}
for result in self.results:
for eval_name, score in result["scores"].items():
if eval_name not in all_scores:
all_scores[eval_name] = []
if isinstance(score, (int, float)):
all_scores[eval_name].append(score)
elif isinstance(score, dict) and "f1" in score:
all_scores[eval_name].append(score["f1"])
summary = {
"total_tests": len(self.results),
"average_scores": {
name: sum(scores) / len(scores)
for name, scores in all_scores.items()
if scores
},
"total_token_cost": sum(
r["tokens"]["input"] + r["tokens"]["output"]
for r in self.results
)
}
return summaryTest Suite: Format
The test suite format stays intentionally minimal: an input, the expected output, and free-form metadata. This metadata is more useful than it appears—by tagging each case by category, you break down scores and discover that your classifier isn’t failing at random but on a specific family of requests, which a global average of 88% doesn’t reveal. The three examples below also show what a good suite should contain: clear cases, like a startup crash, but also cases that border another category—a question about a missing feature looks a lot like a bug report, and it’s on this boundary that a classifier’s quality is decided.
[
{
"input_text": "My package hasn't arrived",
"expected_output": "complaint",
"metadata": {"category": "delivery"}
},
{
"input_text": "How do I enable dark mode?",
"expected_output": "question",
"metadata": {"category": "feature"}
},
{
"input_text": "The app crashes on startup on iOS 18",
"expected_output": "bug",
"metadata": {"category": "technical"}
}
]CI/CD Integration
Run evaluation on every prompt change. The ci_eval_check function turns measurement into a guardrail: it runs the pipeline, compares the average score to a threshold, and returns a boolean that your integration chain interprets as a build success or failure. A change that drops precision below 85% no longer reaches production, just as a broken unit test blocks a merge.
Set the threshold based on the score already achieved by the current version, not from an ideal. Set it too high and you condemn the team to bypass verification with temporary disablements that become permanent; set it too low and you’re checking nothing while displaying a green badge. Finally, keep in mind that the suite must remain representative of real traffic: a threshold met on twenty cases written two years ago proves nothing about today’s requests.
def ci_eval_check(prompt_config: dict,
test_suite: str,
min_score: float = 0.85) -> bool:
"""CI/CD gate: checks that the prompt meets the minimum score."""
pipeline = EvalPipeline(test_suite)
def accuracy(question, prediction, reference):
return 1.0 if prediction.strip().lower() == reference.strip().lower() else 0.0
results = pipeline.run(prompt_config, [accuracy])
avg_score = results["average_scores"].get("accuracy", 0)
print(f"Average score: {avg_score:.2%}")
print(f"Minimum threshold: {min_score:.2%}")
if avg_score < min_score:
print("FAILURE: score below threshold")
return False
print("SUCCESS: prompt validated")
return TrueEvaluation Dashboard
Track score evolution over time. A single score says nothing; the series tells your prompt’s story and answers the question that always comes up in meetings: are we making progress? The function below produces a Markdown table placing side by side, for each version, the average score, tokens consumed, and estimated cost. The format isn’t accidental: it pastes directly into a pull request, making comparison readable for people who will never open your code.
def comparison_report(results: dict[str, dict]) -> str:
"""Generates a comparative report between prompt versions."""
report = "# Comparative Evaluation Report\n\n"
report += "| Version | Average Score | Average Tokens | Cost |\n"
report += "|---------|---------------|----------------|------|\n"
for version, data in results.items():
score = data.get("average_scores", {})
score_val = list(score.values())[0] if score else 0
tokens = data.get("total_token_cost", 0)
report += (f"| {version} | {score_val:.2%} | "
f"{tokens} | ${tokens * 5 / 1e6:.4f} |\n")
return reportSet Up Your Guardrails
Build a suite of twenty test cases for a real prompt, placing a handful of edge cases rather than twenty comfortable examples: a suite that passes at 100% on the first try teaches you nothing. Implement the three evaluator types and run them together on two versions of your prompt. Exact match gives a clean number, semantic similarity catches legitimate rewordings, and the judge illuminates cases where the first two disagree—these disagreements are richest in lessons about what you actually expect from the model.
Compare, identify the winning version, then set up the CI script that blocks deployment if the score drops. You’ll know the system works the day it rejects a change you thought was obvious.
Key Takeaways
- Automated evaluation is essential for iterating with confidence
- Combine exact, semantic, and LLM-as-judge evaluation as needed
- Maintain a test suite that evolves with your application
- Integrate evaluation into your CI/CD with a minimum threshold
- Use a more capable model as judge to evaluate a cheaper model
- Document each version’s scores to track evolution
Test Your Knowledge
Advanced techniques and industrialization: the final check.
1. What does an effective system prompt contain?
Answer: Role, scope, behavior rules, output format, and edge case handling—written like a specification, tested like code.
2. When should you use self-consistency?
Answer: On reasoning problems with verifiable answers: generate multiple independent reasonings and keep the majority answer—variance becomes an ally.
3. What does JSON mode with schema enforcement guarantee?
Answer: Output strictly conforming to the schema (types, required fields, enums)—validated on the API side and revalidated in code (JSON Schema, Zod) for a surprise-free chain.
4. What is grounding?
Answer: Anchoring responses in provided facts (documents, retrieved data) rather than the model’s memory—with the requirement to cite or abstain outside sources.
5. What is the purpose of a versioned prompt catalog?
Answer: To treat prompts as artifacts: versions, A/B tests, automated evaluations on every change—you know what’s running in production, and why.
The final loop of the course—version, test, evaluate automatically—transforms artisanal prompt engineering into a team practice.