Iterating: When to Re-Fine-Tune

Update November 2, 2026 — Fine-tuning availability. OpenAI’s self-service fine-tuning is being phased out: since May 7, 2026, organizations that have never run a fine-tuning job can no longer create one; since July 2, you must have used a fine-tuned model within the last 60 days; and starting January 6, 2027, no customer will be able to launch new jobs. Already trained models remain usable until their base model is retired. This course remains useful for understanding and leveraging an existing model; for new projects, prioritize prompt engineering, RAG, and evaluations.

Fine-tuning is an iterative process

Your first fine-tuning run will rarely be perfect. Improvement comes from iterative cycles: analyze errors, enrich your data, retrain, and re-evaluate. This lesson teaches you how to identify when and how to iterate effectively—and, just as importantly, how to recognize situations where retraining would be pointless.

Analyze errors before fixing them

The natural reflex after a disappointing evaluation is to randomly add examples, hoping volume will solve the problem. That’s the fastest way to burn your training budget without making progress. Start by sorting each failure into a category: incorrect formatting, inappropriate tone, factual error, incomplete response, off-topic answer, or pure hallucination. The resulting distribution tells you where to focus your effort, and it often reveals a surprise—what you thought was a style issue turns out to be a completeness problem.

def analyze_errors(results: list[dict]) -> dict:
    """Categorizes errors from the fine-tuned model."""
    categories = {
        "format": [],       # Incorrect response format
        "tone": [],         # Inappropriate tone
        "factual": [],      # Incorrect information
        "incomplete": [],   # Response too short or missing elements
        "off_topic": [],    # Does not answer the question
        "hallucination": [] # Invents information
    }

    for r in results:
        if r.get("score", 5) < 3:
            # Classify the error (can be done manually or by an LLM)
            category = classify_error(r)
            categories[category].append(r)

    print("Error distribution:")
    for cat, errors in categories.items():
        if errors:
            print(f"  {cat}: {len(errors)} cases")

    return categories

Next, review cases within the same category to find a common denominator. Ask yourself the same questions each time: does the model fail on a specific type of question, do errors cluster around certain topics, and does the problem stem from the data or the hyperparameters? Failures scattered across all topics but always of the same nature point to a lack of examples; failures concentrated in one domain indicate a coverage gap; and responses that stick too closely to your training examples regardless of the question signal overfitting that needs correction on the epochs side.

Recognizing when to iterate

Certain signals unambiguously call for a new version, and each requires a different response. A model that hallucinates on a topic isn’t fixed by showing it more correct answers on that topic, but by teaching it to politely decline when it doesn’t know.

Conversely, several situations should stop you from retraining. When errors are rare and non-systematic, a prompt adjustment handles them faster and cheaper. When complaints concern latency or wait times, the problem lies in your infrastructure, and no amount of training will fix it. When you’ve collected only a handful of new examples since the last version, the expected gain won’t cover the cost of the run. And when a new base model becomes available, test it as-is first: sometimes it achieves on its own, without training, what your previous version struggled to deliver.

Enrich rather than start from scratch

Incremental iteration means adding targeted examples to an existing dataset without touching the rest. This is the most frequent and simplest operation: you load the current file, concatenate the new cases, and rewrite everything in JSONL format. The final count serves as your control—a total that doesn’t match the expected sum indicates a file read error or lost lines.

import json

def enrich_dataset(
    existing_dataset: str,
    new_examples: list[dict],
    output: str
):
    """Adds new examples to the existing dataset."""
    # Load existing data
    with open(existing_dataset, "r") as f:
        existing = [json.loads(l) for l in f]

    # Merge
    combined = existing + new_examples

    # Save
    with open(output, "w") as f:
        for ex in combined:
            f.write(json.dumps(ex, ensure_ascii=False) + "\n")

    print(f"Enriched dataset: {len(existing)} + {len(new_examples)} = {len(combined)}")

Corrective iteration is even more valuable because it turns your failures into raw material. Every error spotted in production provides a real question; you just need to write the response you wish you’d seen, and the pair becomes a training example. Note that the bad response doesn’t go into the final example: it helps document the case and decide whether the correction is worth the effort, but the model learns only from the correct version.

def error_to_example(
    question: str,
    bad_response: str,
    good_response: str,
    system_msg: str
) -> dict:
    """Transforms an error into a training example."""
    return {
        "messages": [
            {"role": "system", "content": system_msg},
            {"role": "user", "content": question},
            {"role": "assistant", "content": good_response}
        ]
    }

# Transform a batch of errors
corrections = []
for error in detected_errors:
    correction = error_to_example(
        question=error["question"],
        bad_response=error["model_response"],
        good_response=error["corrected_response"],
        system_msg=SYSTEM_PROMPT
    )
    corrections.append(correction)

Continuous fine-tuning and its limits

Nothing stops you from fine-tuning an already fine-tuned model: just pass its full ID in the model parameter instead of the base model name. The operation is tempting since it trains only on the new data.

# Fine-tune from an already fine-tuned model
job_v2 = client.fine_tuning.jobs.create(
    training_file=new_file.id,
    model="ft:gpt-5.6-luna:org::suffix-v1:xxxxxxxx",  # v1 model as base
    suffix="my-assistant-v2"
)

Warning: continuous fine-tuning accumulates biases. After 2–3 successive iterations, it’s often better to start fresh from the base model with a complete, enriched dataset. Each layer amplifies the quirks of the previous one, including those you didn’t intend, and the model ends up responding with a rigidity nobody explicitly taught it.

The full cycle

The steps below form the loop you’ll repeat for each version. If you tackle the most frequent errors first, the next version will show measurable gains on your test set; tackling them out of order yields only diffuse improvement that’s hard to defend to your team.

Document every iteration

After four or five versions, no one remembers what distinguished v2 from v3 or why v4 was abandoned. Systematically recording each run—the dataset composition, the source of new examples, hyperparameters, results, and the decision made—gives you this information when a regression forces you to retrace your steps. Pay special attention to the source of new examples: only that will tell you, months later, whether the degradation came from a poorly chosen data batch.

iteration_log = {
    "version": "v3",
    "date": "2026-04-02",
    "base_model": "gpt-5.6-luna",
    "dataset": {
        "train": 350,
        "validation": 45,
        "new_examples": 50,
        "new_examples_source": "production errors week 12"
    },
    "hyperparameters": {
        "n_epochs": 3,
        "learning_rate_multiplier": "auto",
        "batch_size": "auto"
    },
    "results": {
        "validation_loss": 0.42,
        "precision_test": 0.91,
        "avg_csat": 4.3,
        "improvement_vs_v2": "+5% precision, +0.2 CSAT"
    },
    "decision": "Deployed to production on 2026-04-03"
}

Key takeaways

• Fine-tuning is iterative—plan for multiple cycles
• Categorize errors before fixing them in the data
• Turn user feedback into training examples
• After 2–3 successive iterations, restart from the base model
• Document every iteration to maintain a clear history