A/B Testing Prompts
On this page
Lesson updated November 3, 2026 based on OpenAI documentation:
temperatureandtop_pare only accepted with a reasoning effort ofnone, whereas GPT-5.6 models reason by default (medium). Calls that set these parameters must therefore specifyreasoning={"effort": "none"}(Responses API) orreasoning_effort="none"(Chat Completions). Sources: Using GPT-6 · GPT-5.6 Terra.
A/B Testing Prompts
A/B testing prompts lets you objectively compare two versions of a prompt using real traffic. Instead of choosing a prompt based on intuition, you measure which one produces better results according to your business metrics. This is the mandatory step between prototyping and production.
Principle
Incoming traffic is split randomly between two (or more) prompt variants. Each response is evaluated against predefined criteria, and the winning variant is deployed to 100% of the traffic.
The strength of this method is that both variants face the same traffic, under the same conditions, at the same time. All external sources of variation—seasonality, network incidents, the arrival of a new category of requests—affect both groups equally and cancel out in the comparison. A sequential test cannot guarantee this: running the old prompt for a week and the new one for the next leaves you unable to tell whether the improvement came from your rewrite or the calendar.
A/B Testing Framework
The PromptExperiment class has three responsibilities. assign_variant decides which variant serves a given request, run executes the call and logs the result, and get_stats aggregates the data.
Assignment deserves a note. When a user_id is provided, it is deterministic, calculated from a hash of the experiment-user pair. The same user stays in the same group from one call to the next. This continuity is not a theoretical refinement: a customer who receives a warm response and then, three minutes later, a dry response to their follow-up will think it’s a bug, and their dissatisfaction will pollute both variants.
Each execution records the input, output, tokens consumed, and timestamp. These tokens are not there for decoration: a variant might win on quality while doubling the response length, and _estimate_cost translates that gap into dollars so you can make an informed trade-off.
import random
import json
from datetime import datetime
from openai import OpenAI
client = OpenAI()
class PromptExperiment:
"""A/B testing framework for prompts."""
def __init__(self, name: str, variants: dict[str, dict]):
"""
variants = {
"control": {"system_prompt": "...", "model": "gpt-5.6-terra",
"temperature": 0.3},
"variant_a": {"system_prompt": "...", "model": "gpt-5.6-terra",
"temperature": 0.1},
}
"""
self.name = name
self.variants = variants
self.results = {v: [] for v in variants}
def assign_variant(self, user_id: str = None) -> str:
"""Assigns a variant (sticky by user_id if provided)."""
if user_id:
# Deterministic assignment by user
hash_val = hash(f"{self.name}:{user_id}") % 100
keys = sorted(self.variants.keys())
split = 100 // len(keys)
for i, key in enumerate(keys):
if hash_val < (i + 1) * split:
return key
return keys[-1]
return random.choice(list(self.variants.keys()))
def run(self, input_text: str, user_id: str = None,
schema: dict = None) -> dict:
"""Executes the prompt with the assigned variant."""
variant_name = self.assign_variant(user_id)
variant = self.variants[variant_name]
kwargs = {
"model": variant["model"],
"instructions": variant["system_prompt"],
"input": input_text,
"temperature": variant.get("temperature", 0.3)
}
if schema:
kwargs["text"] = {
"format": {
"type": "json_schema",
"name": "response",
"schema": schema,
"strict": True
}
}
response = client.responses.create(**kwargs)
result = {
"variant": variant_name,
"input": input_text,
"output": response.output_text,
"tokens_input": response.usage.input_tokens,
"tokens_output": response.usage.output_tokens,
"timestamp": datetime.now().isoformat()
}
self.results[variant_name].append(result)
return result
def get_stats(self) -> dict:
"""Calculates statistics per variant."""
stats = {}
for variant, results in self.results.items():
if not results:
continue
tokens_in = [r["tokens_input"] for r in results]
tokens_out = [r["tokens_output"] for r in results]
stats[variant] = {
"count": len(results),
"avg_tokens_input": sum(tokens_in) / len(tokens_in),
"avg_tokens_output": sum(tokens_out) / len(tokens_out),
"total_cost_estimate": self._estimate_cost(
sum(tokens_in), sum(tokens_out)
)
}
return stats
def _estimate_cost(self, tokens_in: int,
tokens_out: int) -> float:
"""Estimates cost in USD."""
return (tokens_in * 2.5 + tokens_out * 10) / 1_000_000Evaluation Metrics
Metrics depend on your use case, and two families cover most scenarios. When the task has a correct answer—classifying a ticket, extracting a field—evaluation boils down to comparing against a reference label. This is the role of evaluer_classification. Note an instructive choice here: parsing failures are treated as empty predictions rather than raising an exception, because a variant that produces invalid JSON should be penalized in the score, not crash the measurement.
When the task produces free text, there is no single correct answer, so you ask the model to arbitrate. evaluer_qualite_texte breaks the judgment down into four criteria—relevance, clarity, completeness, professional tone—rather than asking for a direct overall score. This breakdown makes evaluation more stable and tells you on which axis the winning variant excels: knowing the gain comes from tone rather than completeness is precisely the information you need to write the next version.
def evaluer_classification(result: dict,
label_attendu: str) -> dict:
"""Evaluates a classification response."""
try:
output = json.loads(result["output"])
prediction = output.get("category", "")
except json.JSONDecodeError:
prediction = ""
return {
"correct": prediction == label_attendu,
"prediction": prediction,
"attendu": label_attendu
}
def evaluer_qualite_texte(result: dict) -> dict:
"""Evaluates the quality of a text response using the LLM."""
eval_response = client.responses.create(
model="gpt-5.6-terra",
reasoning={"effort": "none"}, # temperature requires effort none
instructions="You are a text quality evaluator. "
"Rate out of 10 based on: relevance, clarity, "
"completeness, professional tone.",
input=f"Question: {result['input']}\n\n"
f"Answer: {result['output']}",
text={
"format": {
"type": "json_schema",
"name": "evaluation",
"schema": {
"type": "object",
"properties": {
"pertinence": {"type": "integer"},
"clarte": {"type": "integer"},
"completude": {"type": "integer"},
"ton": {"type": "integer"},
"note_globale": {"type": "number"},
"commentaire": {"type": "string"}
},
"required": ["pertinence", "clarte",
"completude", "ton",
"note_globale", "commentaire"],
"additionalProperties": False
},
"strict": True
}
},
temperature=0.1
)
return json.loads(eval_response.output_text)Complete Example: Testing Two Prompt Styles
The experiment below contrasts two approaches to customer support. The control uses a minimal, professional, and neutral instruction. The “empathetic” variant imposes a three-step structure: acknowledge the problem, express empathy, then propose a solution with steps. Everything else is identical, from the model to the temperature. This symmetry is the condition for attributing any measured gap to the prompt. Changing both tone and temperature together would yield an uninterpretable result.
The five test tickets deliberately cover different registers: an emotionally charged delivery incident, a functional bug, a neutral password question, a contentious refund request, and an API outage reported by a technical profile. You expect the empathetic variant to dominate the first two and be penalized on the last, where the interlocutor wants a status and an ETA, not compassion. This is the nuance that a global average masks and that a ticket-by-ticket examination reveals—often leading to the real decision: route based on request type rather than choosing a single tone.
experiment = PromptExperiment(
name="customer-support-style",
variants={
"control": {
"system_prompt": "You are a support agent. "
"Respond professionally.",
"model": "gpt-5.6-terra",
"temperature": 0.3
},
"empathique": {
"system_prompt": "You are a caring support agent. "
"Start by acknowledging the customer's problem, "
"show empathy, then propose a concrete solution "
"with steps.",
"model": "gpt-5.6-terra",
"temperature": 0.3
}
}
)
# Simulate traffic
test_tickets = [
"My package hasn't arrived in 2 weeks.",
"The app crashes when I click on Settings.",
"How do I change my password?",
"I want a refund, the product is defective.",
"Your API has been returning a 500 error since this morning.",
]
for ticket in test_tickets:
result = experiment.run(ticket)
quality = evaluer_qualite_texte(result)
print(f"Variant: {result['variant']} | "
f"Score: {quality['note_globale']}/10")
print("\nStatistics:")
print(json.dumps(experiment.get_stats(), indent=2))Statistical Significance
Don’t draw conclusions too quickly. With little data, results are noisy, and the classic mistake is seeing 7.4 versus 6.9 over twelve observations and declaring a winner. The Student’s t-test below answers the only question that matters: is the observed gap larger than what chance would naturally produce between two equivalent series?
A p_value below 0.05 means such a gap would be unlikely if the two variants were equal. Above that, you have demonstrated nothing—which is not the same as demonstrating equality. Most often, you just lack data. Count on a minimum of one hundred observations per variant before concluding, and beware the urge to check results every morning. If you look often enough, you will always find a moment where the gap seems meaningful.
from scipy import stats
def test_significativite(scores_a: list[float],
scores_b: list[float]) -> dict:
"""Student's t-test to compare two variants."""
t_stat, p_value = stats.ttest_ind(scores_a, scores_b)
return {
"moyenne_a": sum(scores_a) / len(scores_a),
"moyenne_b": sum(scores_b) / len(scores_b),
"p_value": p_value,
"significatif": p_value < 0.05,
"recommandation": (
"Significant difference, deploy the winning variant"
if p_value < 0.05
else "Not enough data or no significant difference"
)
}Running Your First Experiment
Take a task you know well and set up a PromptExperiment with two variants, changing only one thing between the control and the variant: a tone, an imposed structure, or an additional instruction. Define two or three metrics before launching anything, because choosing the metric after seeing the results is like picking the winner in advance, and it is the most common bias in homegrown tests.
Run the experiment on twenty inputs, then analyze. Look at the average, but especially open the cases where the two variants diverge the most: these explain the gap. Twenty observations will obviously not be enough to reach significance, and realizing this yourself is better than reading it here. Finally, document the decision and archive the raw results with the date and exact variant configuration—this archive is what will stop you from running the same experiment again in a year.
Key Takeaways
- A/B testing replaces intuition with measurable data
- Assign variants deterministically by user
- Measure business metrics, not just perceived quality
- Wait for statistical significance before concluding
- Document every experiment and its results