Evaluate Your Fine-Tuned Model

Update November 2, 2026 — Fine-tuning availability. OpenAI’s self-service fine-tuning is being phased out: since May 7, 2026, organizations that have never used it can no longer create jobs; since July 2, you must have used a fine-tuned model within the last 60 days; and starting January 6, 2027, no customer will be able to launch new jobs. Already trained models remain usable until their base model is retired. This course remains useful for understanding and leveraging an existing model; for new projects, prioritize prompt engineering, RAG, and evaluations.

Why Evaluation Is Essential

A successful fine-tune isn’t measured by loss alone. The real question is: is your fine-tuned model actually better than the base model for your use case? A cleanly descending curve only proves the model learned to reproduce your examples; it says nothing about its behavior on unseen requests. This lesson shows you how to build a rigorous evaluation protocol, combining automated metrics, LLM judging, and human review.

Building a Test Set

Your test set must be separate from both your training and validation data. The distinction matters: the validation set monitors training and therefore influences your hyperparameter decisions, which gradually contaminates it. The test set, by contrast, is opened only once the model is frozen. It represents the real-world situations your model will encounter in production, with the same proportions of simple and tricky cases. An 80/10/10 split on a previously shuffled corpus is a reasonable starting point.

import json
import random

def prepare_datasets(file: str, ratios: tuple = (0.8, 0.1, 0.1)):
    """Splits data into train/validation/test."""
    with open(file, "r") as f:
        examples = [json.loads(l) for l in f]

    random.shuffle(examples)
    n = len(examples)
    i1 = int(n * ratios[0])
    i2 = int(n * (ratios[0] + ratios[1]))

    datasets = {
        "train": examples[:i1],
        "validation": examples[i1:i2],
        "test": examples[i2:]
    }

    for name, data in datasets.items():
        with open(f"{name}.jsonl", "w") as f:
            for ex in data:
                f.write(json.dumps(ex, ensure_ascii=False) + "\n")
        print(f"{name}: {len(data)} examples")

    return datasets

prepare_datasets("all_data.jsonl")

Automated Evaluation

Metric-Based Evaluation

The principle is simple: for each test example, remove the assistant message—that’s the expected answer, which the model must not see—submit the rest, and keep the input, reference answer, and generated answer side by side. This three-column structure is the foundation for everything that follows; using it for metric calculation or qualitative judgment is just a matter of downstream processing.

from openai import OpenAI

client = OpenAI()

def evaluate_model(model: str, test_examples: list[dict]) -> dict:
    """Evaluates a model on a test set."""
    results = []

    for ex in test_examples:
        # Extract the question (without the expected answer)
        input_messages = [
            m for m in ex["messages"] if m["role"] != "assistant"
        ]
        expected_answer = next(
            m["content"] for m in ex["messages"] if m["role"] == "assistant"
        )

        # Get the model's response
        response = client.responses.create(
            model=model,
            input=input_messages
        )
        model_answer = response.output_text

        results.append({
            "input": input_messages[-1]["content"],
            "expected": expected_answer,
            "obtained": model_answer
        })

    return results

LLM-as-a-Judge Evaluation

A powerful technique is to use a high-performing model to judge response quality. Word-for-word comparison with the reference becomes unusable as soon as multiple phrasings are acceptable; a judge, however, appreciates the substance. The quality of its verdict depends entirely on the precision of the prompt: four criteria rated 1 to 5, a forced JSON output, and a free-text comment you’ll use later to understand failures. Plan for cases where the output isn’t parseable rather than letting an exception interrupt the campaign.

def evaluate_with_judge(results: list[dict]) -> list[dict]:
    """Uses gpt-5.6-sol as a judge to evaluate responses."""
    evaluations = []

    for r in results:
        judge_prompt = f"""Evaluate the quality of the obtained response compared to the expected response.

Question: {r['input']}

Expected response: {r['expected']}

Obtained response: {r['obtained']}

Criteria (rate each from 1 to 5):
1. Accuracy: Is the information correct?
2. Completeness: Are all important points covered?
3. Style: Is the tone and format appropriate?
4. Conciseness: Is the response an appropriate length?

Respond in JSON: {{"accuracy": N, "completeness": N, "style": N, "conciseness": N, "comment": "..."}}"""

        response = client.responses.create(
            model="gpt-5.6-sol",
            input=judge_prompt
        )

        try:
            evaluation = json.loads(response.output_text)
            evaluations.append(evaluation)
        except json.JSONDecodeError:
            evaluations.append({"error": "Response not parseable"})

    return evaluations

Specialized Metrics

The two previous approaches work well for written responses. As soon as your model’s output has a fixed structure, more direct measures become possible and save you the detour through a judge.

For Classification

When the expected output belongs to a closed set—a ticket category, a sentiment, a product code—exact match becomes relevant again. Normalize case and spaces before comparing; otherwise, “Urgent” versus “urgent” will count as an error and skew your rate.

def classification_metrics(results: list[dict]) -> dict:
    """Calculates precision, recall, and F1 for a classification task."""
    correct = sum(
        1 for r in results
        if r["expected"].strip().lower() == r["obtained"].strip().lower()
    )
    total = len(results)
    precision = correct / total if total > 0 else 0

    print(f"Precision: {precision:.2%} ({correct}/{total})")
    return {"precision": precision, "correct": correct, "total": total}

For Function Calling

Function calling is judged on two axes that must be measured separately. A model that picks the right function but gets the parameter wrong fails differently from one that calls the completely wrong tool: the first problem is fixed by extraction examples, the second by discrimination examples. A single global score would mask this distinction and leave you working blind.

def function_calling_metrics(results: list[dict]) -> dict:
    """Evaluates the quality of function calls."""
    stats = {"correct_function": 0, "correct_params": 0, "total": 0}

    for r in results:
        stats["total"] += 1
        if r.get("expected_function") == r.get("obtained_function"):
            stats["correct_function"] += 1
        if r.get("expected_params") == r.get("obtained_params"):
            stats["correct_params"] += 1

    print(f"Correct function: {stats['correct_function']}/{stats['total']}")
    print(f"Correct parameters: {stats['correct_params']}/{stats['total']}")
    return stats

Human Evaluation

Automated evaluation doesn’t capture everything. A slightly condescending tone, a phrase that doesn’t sit well with your customers, a technically correct but unusable answer: these flaws only appear under a human eye. The protocol is straightforward. Sample twenty to fifty responses at random, never the cases you’ve already flagged as problematic, or you’ll just be measuring your own selection bias. Then define clear, measurable criteria and formalize them in a standardized scoring rubric so two reviewers judge the same thing.

Finally, have multiple people evaluate and calculate inter-rater agreement. Strong disagreement doesn’t necessarily signal a bad model: most often, it reveals that your rubric remains ambiguous and needs clarification before you can conclude anything.

Sample Evaluation Rubric

Evaluation Report

One final reflex separates an evaluation from a decision: record both families of results in a single report, accompanied by an explicit recommendation. The threshold you set—here a human average of 4 to go to production, 3 to iterate on data—must be chosen before launching the campaign. Decided afterward, it always adjusts to the result you hoped for.

def evaluation_report(auto_results: dict, human_results: dict):
    """Generates a combined evaluation report."""
    print("=" * 60)
    print("FINE-TUNED MODEL EVALUATION REPORT")
    print("=" * 60)

    print(f"\nAutomated metrics:")
    for k, v in auto_results.items():
        print(f"  {k}: {v}")

    print(f"\nHuman evaluation:")
    for k, v in human_results.items():
        print(f"  {k}: {v}")

    print(f"\nRecommendation:")
    average_score = sum(human_results.values()) / len(human_results)
    if average_score >= 4:
        print("  Model is ready for production")
    elif average_score >= 3:
        print("  Improvements needed—iterate on data")
    else:
        print("  Insufficient results—rethink your approach")

Key Takeaways

• Always separate an independent test set (neither train nor validation)
• Combine automated and human evaluation
• LLM-as-a-judge evaluation is a good compromise between the two
• Define measurable criteria before you start
• A low loss score doesn’t guarantee production quality