A/B Testing and Business Metrics
On this page
Update November 2, 2026 — Fine-tuning availability. OpenAI’s self-service fine-tuning is being phased out: since May 7, 2026, organizations that have never used it can no longer create jobs; since July 2, you must have used a fine-tuned model within the last 60 days; and starting January 6, 2027, no customer will be able to launch new jobs. Already trained models remain usable until their base model is retired. This course remains useful for understanding and leveraging an existing model; for new projects, prioritize prompt engineering, RAG, and evaluations.
Beyond Loss: Measuring Business Impact
Training loss and quality scores aren’t enough. What really matters is the impact of your fine-tuned model on your business KPIs: customer satisfaction, resolution time, conversion rate. A model might produce responses your reviewers find elegant while simultaneously increasing the escalation rate to your human agents. That second number decides the project’s future, not the first.
Setting Up an A/B Test
An A/B test compares two variants under real-world conditions with authentic traffic. Group A, the control group, receives responses from the base model with your optimized prompt; Group B, the test group, gets responses from the fine-tuned model. Users are randomly assigned to both groups, and you measure results using the same metrics. The entire validity of the exercise hinges on this assignment: if Group B happens to receive the simplest requests by chance, its apparent superiority measures nothing but that luck.
The choisir_variante function solves this problem by hashing the user ID. This approach guarantees that the same user always sees the same variant—essential, because alternating between sessions would expose the person to two response styles and skew their satisfaction rating. It produces a statistically balanced distribution once volume is sufficient, and it remains reproducible: rerunning the calculation later yields exactly the same groups, letting you recalculate a forgotten metric without relaunching the test.
import random
import hashlib
from openai import OpenAI
client = OpenAI()
MODELE_BASE = "gpt-5.6-luna"
MODELE_FT = "ft:gpt-5.6-luna:org::suffix:xxxxxxxx"
PROMPT_BASE = "You are a professional customer support assistant..."
def choisir_variante(user_id: str, pourcentage_test: int = 50) -> str:
"""Assigns a user to group A or B deterministically."""
h = int(hashlib.md5(user_id.encode()).hexdigest(), 16)
return "B" if (h % 100) < pourcentage_test else "A"
def repondre(user_id: str, question: str) -> dict:
"""Responds with the correct model based on the A/B group."""
variante = choisir_variante(user_id)
if variante == "A":
response = client.responses.create(
model=MODELE_BASE,
instructions=PROMPT_BASE,
input=question
)
else:
response = client.responses.create(
model=MODELE_FT,
input=question
)
return {
"variante": variante,
"modele": MODELE_BASE if variante == "A" else MODELE_FT,
"reponse": response.output_text,
"user_id": user_id
}Choosing the Right Business Metrics
The indicators to track depend entirely on what your assistant is supposed to deliver, and each use case family has its own.
For a customer support service, the first-contact resolution rate answers the most concrete question: did the user need to contact support again? The CSAT score, a 1-to-5 satisfaction rating collected after the interaction, captures sentiment that automated metrics miss. Resolution time measures the total duration of the exchange, and the escalation rate—the portion of conversations transferred to a human—reads as an admission of model failure on the cases involved. Successful fine-tuning typically lowers both of the latter simultaneously.
For a sales chatbot, the reasoning shifts downstream. The conversion rate counts conversations leading to a purchase or appointment; the engagement rate, expressed as the average number of messages per conversation, requires careful interpretation since a longer exchange could signal either interest or misunderstanding; the bounce rate isolates users who leave after the first message, often because the initial response missed the mark.
For a classification or extraction task, you return to the classic trio. Precision gives the percentage of correct classifications, recall the percentage of positive cases actually detected, and the F1-score, the harmonic mean of both, prevents settling for a model that never predicts the rare class just to avoid errors.
Collecting Measurements
These indicators are useless if they live in three separate spreadsheets. A single collector, fed at every interaction, keeps the variant alongside the metric and enables producing a comparative report with a single command. The ABTracker class below does this minimal work: it logs, then groups by variant and calculates averages and rates.
import datetime
class ABTracker:
"""Metric collector for A/B testing."""
def __init__(self):
self.events = []
def log_interaction(
self,
user_id: str,
variante: str,
question: str,
reponse: str,
metriques: dict
):
"""Records an interaction with its metrics."""
self.events.append({
"timestamp": datetime.datetime.now().isoformat(),
"user_id": user_id,
"variante": variante,
"question": question,
"reponse": reponse,
**metriques
})
def rapport(self) -> dict:
"""Generates a comparative A vs B report."""
groupes = {"A": [], "B": []}
for e in self.events:
groupes[e["variante"]].append(e)
rapport = {}
for variante, events in groupes.items():
if not events:
continue
rapport[variante] = {
"nb_interactions": len(events),
"csat_moyen": self._moyenne(events, "csat"),
"resolution_premier_contact": self._taux(events, "resolu"),
"temps_moyen_sec": self._moyenne(events, "temps_resolution"),
}
return rapport
def _moyenne(self, events, cle):
vals = [e[cle] for e in events if cle in e and e[cle] is not None]
return sum(vals) / len(vals) if vals else None
def _taux(self, events, cle):
vals = [e[cle] for e in events if cle in e]
return sum(vals) / len(vals) * 100 if vals else None
tracker = ABTracker()Waiting for Statistical Significance
Don’t draw conclusions too early. After twenty interactions per group, a 0.4-point CSAT gap has no value: three happy customers are enough to produce it. You need enough interactions for results to be significant, and the benchmarks below give you the order of magnitude to aim for before even looking at the numbers.
Once you reach that volume, a statistical test decides for you. The Mann-Whitney test works well for satisfaction scores, which are ordinal ratings rarely normally distributed. Its p-value answers a precise question: what is the probability of observing such a large gap if the two models were actually equal? Below 0.05, the hypothesis of simple chance becomes hard to defend.
from scipy import stats
def test_significativite(scores_a: list[float], scores_b: list[float]):
"""Tests whether the difference between A and B is significant."""
stat, p_value = stats.mannwhitneyu(scores_a, scores_b, alternative="two-sided")
moy_a = sum(scores_a) / len(scores_a)
moy_b = sum(scores_b) / len(scores_b)
print(f"Mean A: {moy_a:.3f}")
print(f"Mean B: {moy_b:.3f}")
print(f"p-value: {p_value:.4f}")
if p_value < 0.05:
gagnant = "B (fine-tuned)" if moy_b > moy_a else "A (base)"
print(f"Significant difference — {gagnant} is better")
else:
print("No significant difference")Scaling Up Gradually
Even a successful test doesn’t justify switching all traffic overnight: a flaw that appears only once in a thousand remains invisible on a sample and becomes very visible at full volume. Proceed in one-week stages, checking your metrics at each step before opening the next.
- Phase 1 (1 week): 10% traffic on the fine-tuned model
- Phase 2 (1 week): 30% if metrics are positive
- Phase 3 (1 week): 50% for the main A/B test
- Phase 4: 100% if the test is conclusive
If a stage degrades your metrics, revert to the previous one instead of opening the next hoping the number will recover on its own. A week spent understanding why always costs less than a month of traffic served by a disappointing model.
Key Takeaways
- Business metrics matter more than training loss
- Use deterministic hashing for A/B assignment
- Wait for statistical significance before concluding (p < 0.05)
- Deploy gradually: 10%, 30%, 50%, 100%
- Measure costs and latency in addition to quality