Compare with the Base Model

Update November 2, 2026 — Fine-tuning availability. OpenAI’s self-service fine-tuning is being phased out: as of May 7, 2026, organizations that have never used it can no longer create jobs; since July 2, you must have used a fine-tuned model within the last 60 days; and starting January 6, 2027, no customer will be able to launch new jobs. Already trained models remain usable until their base model is retired. This course remains useful for understanding and leveraging an existing model; for new projects, prioritize prompt engineering, RAG, and evaluations.

Why systematic comparison matters

Fine-tuning has a cost: data preparation time, training expenses, and maintenance complexity. You must prove that your fine-tuned model delivers a measurable improvement over the base model with a strong prompt. If the difference is marginal, fine-tuning may not be worth the effort.

The temptation in this exercise is to compare your trained model against a poorly instructed base model. That makes for a flattering but worthless demonstration: you’ve beaten an opponent you weakened yourself. The only honest comparison pits your fine-tuned model against the best possible use of the base model.

Define the baseline

The baseline is the base model with your best system prompt. Optimize this prompt before comparing. Concretely, this means writing down the rules you implicitly taught by example during fine-tuning: the role, tone, expected length, and how to close an exchange. The prompt below does exactly that for a support assistant; note that the fine-tuned model needs no such instructions, since these rules are now part of its weights.

from openai import OpenAI

client = OpenAI()

OPTIMIZED_SYSTEM_PROMPT = """You are a customer support assistant for TechCorp.
Rules:
- Respond professionally and empathetically
- Always propose a concrete solution
- Limit responses to 3 sentences maximum
- End with a follow-up question"""

def baseline_response(question: str) -> str:
    """Gets a response from the base model."""
    response = client.responses.create(
        model="gpt-5.6-luna",
        instructions=OPTIMIZED_SYSTEM_PROMPT,
        input=question
    )
    return response.output_text

def fine_tune_response(question: str, model_ft: str) -> str:
    """Gets a response from the fine-tuned model."""
    response = client.responses.create(
        model=model_ft,
        input=question
    )
    return response.output_text

Generate side-by-side responses

The next step is to query both models on exactly the same questions and keep the paired responses. Choose questions representative of your real traffic rather than textbook cases: a delivery complaint, a procedure request, dissatisfaction, a refund request, a factual question. These five families make the difference in production, not perfect phrasing.

import json

def compare_models(
    questions: list[str],
    base_model: str,
    ft_model: str,
    system_prompt: str
) -> list[dict]:
    """Compares responses from two models on the same questions."""
    comparisons = []

    for q in questions:
        # Base model with optimized prompt
        r_base = client.responses.create(
            model=base_model,
            instructions=system_prompt,
            input=q
        )

        # Fine-tuned model (minimal or no prompt)
        r_ft = client.responses.create(
            model=ft_model,
            input=q
        )

        comparisons.append({
            "question": q,
            "baseline_response": r_base.output_text,
            "ft_response": r_ft.output_text
        })

    return comparisons

# Test questions
test_questions = [
    "My package hasn't arrived.",
    "How do I reset my password?",
    "Your product isn't working as expected.",
    "I would like a refund.",
    "What are your opening hours?",
]

results = compare_models(
    test_questions,
    "gpt-5.6-luna",
    "ft:gpt-5.6-luna:org::suffix:xxxxxxxx",
    OPTIMIZED_SYSTEM_PROMPT
)

Have a third-party model judge

Reading a hundred pairs by hand quickly becomes discouraging, and judgment slips as fatigue sets in. A third-party model, which produced neither response, arbitrates each pair on four criteria and renders a global verdict. First, neutralize the labels: the judge sees “Response A” and “Response B,” never which model produced them, otherwise its verdict will weigh the model name as much as the text. Next, handle cases where the output isn’t parseable; otherwise, a single malformed response halts the entire campaign.

def judge_comparison(comparison: dict) -> dict:
    """Has a third-party model judge the comparison."""
    prompt = f"""Compare these two responses to the same question.

Question: {comparison['question']}

Response A: {comparison['baseline_response']}

Response B: {comparison['ft_response']}

For each criterion, indicate which response is better (A, B, or tie):
1. Relevance of the response
2. Professionalism of tone
3. Adherence to expected format
4. Usefulness to the user

Respond in JSON: {{"relevance": "A/B/tie", "tone": "A/B/tie", "format": "A/B/tie", "usefulness": "A/B/tie", "overall_winner": "A/B/tie"}}"""

    response = client.responses.create(
        model="gpt-5.6-sol",
        input=prompt
    )

    try:
        return json.loads(response.output_text)
    except json.JSONDecodeError:
        return {"error": "Response not parseable"}

# Judge all comparisons
judgments = [judge_comparison(c) for c in results]

Now aggregate these verdicts into a single figure. The improvement rate—the share of pairs where the fine-tuned model wins—is read against predefined thresholds: above 60%, the gain is clear; between 40% and 60%, weigh the gain against the extra cost; below that, your fine-tuning hasn’t proven itself, and the problem almost always lies in the data.

def comparison_score(judgments: list[dict]) -> dict:
    """Calculates the overall comparison score."""
    scores = {"A_wins": 0, "B_wins": 0, "tie": 0}

    for j in judgments:
        winner = j.get("overall_winner", "tie")
        if winner == "A":
            scores["A_wins"] += 1
        elif winner == "B":
            scores["B_wins"] += 1
        else:
            scores["tie"] += 1

    total = len(judgments)
    print(f"Base model wins: {scores['A_wins']}/{total}")
    print(f"Fine-tuned model wins: {scores['B_wins']}/{total}")
    print(f"Tie: {scores['tie']}/{total}")

    improvement_rate = scores["B_wins"] / total * 100
    print(f"\nImprovement rate: {improvement_rate:.1f}%")

    if improvement_rate >= 60:
        print("Fine-tuning provides a significant improvement.")
    elif improvement_rate >= 40:
        print("Modest improvement—check if the cost is justified.")
    else:
        print("Fine-tuning does not provide a clear improvement.")

    return scores

comparison_score(judgments)

Compare costs

Quality is only half the decision. Fine-tuning often reduces input tokens since the long prompt becomes unnecessary, but every token billed on a fine-tuned model costs more. These two effects pull in opposite directions, and only the calculation decides. The function below pits them against each other over a month of traffic: a prompt dropping from 500 to 50 tokens more than offsets a doubled input rate, but that wouldn’t hold true with an already short base prompt.

def compare_costs(
    requests_per_day: int,
    base_prompt_tokens: int,
    ft_prompt_tokens: int,
    avg_output_tokens: int,
    base_input_price: float,
    ft_input_price: float,
    output_price: float
):
    """Compares monthly costs between base and fine-tuned models."""
    days = 30

    base_cost = days * requests_per_day * (
        base_prompt_tokens * base_input_price / 1_000_000 +
        avg_output_tokens * output_price / 1_000_000
    )

    ft_cost = days * requests_per_day * (
        ft_prompt_tokens * ft_input_price / 1_000_000 +
        avg_output_tokens * output_price / 1_000_000
    )

    savings = base_cost - ft_cost
    print(f"Monthly base cost: ${base_cost:.2f}")
    print(f"Monthly fine-tuned cost: ${ft_cost:.2f}")
    print(f"Monthly savings: ${savings:.2f}")

# Example
compare_costs(
    requests_per_day=1000,
    base_prompt_tokens=500,   # long prompt with instructions
    ft_prompt_tokens=50,       # short prompt, behavior is integrated
    avg_output_tokens=200,
    base_input_price=0.15,
    ft_input_price=0.30,        # fine-tuned costs more per token
    output_price=0.60
)

Compare latency

Shortening a prompt by several hundred tokens also affects response time, a criterion your users notice well before tonal finesse. Measure latency on the same questions used for qualitative comparison, outside peak hours, and repeat the measurement several times: a single value mostly reflects the network state at test time.

import time

def measure_latency(model: str, questions: list[str], prompt: str = "") -> float:
    """Measures the average latency of a model."""
    latencies = []
    for q in questions:
        start = time.time()
        kwargs = {"model": model, "input": q}
        if prompt:
            kwargs["instructions"] = prompt
        client.responses.create(**kwargs)
        latencies.append(time.time() - start)

    avg = sum(latencies) / len(latencies)
    print(f"{model} — Average latency: {avg:.2f}s")
    return avg

Decide based on results

Now cross-reference quality and cost. Each combination calls for a different decision, and the most uncomfortable remains the middle ground: when the two models are equal, you must abandon work already paid for rather than seek justification after the fact.

Key takeaways

• Always compare against a base model + optimized prompt, not a basic prompt
• Use LLM judgment to automate qualitative comparison
• Measure costs and latency, not just quality
• An improvement rate of 60%+ justifies fine-tuning
• If fine-tuning doesn’t provide a clear improvement, stick with prompt engineering