Versioning and Rollback

Update November 2, 2026 — Fine-tuning availability. OpenAI’s self-service fine-tuning is being phased out: since May 7, 2026, organizations that have never used it can no longer create jobs; since July 2, you must have used a fine-tuned model within the last 60 days; and starting January 6, 2027, no customer will be able to launch new jobs. Already trained models remain usable until their base model is retired. This course remains useful for understanding and leveraging an existing model; for new projects, prioritize prompt engineering, RAG, and evaluations.

Why version your models

In production, you will iterate on your fine-tuned model: new data, adjusted hyperparameters, new base model. Without a rigorous versioning system, you will lose traceability and the ability to roll back in case of regression. The scenario is always the same: on a Monday morning, user feedback deteriorates, no one knows which model is actually running or what data produced it, and the team spends the day reconstructing information that a few-line file would have preserved.

Naming versions

The suffix parameter of the fine-tuning job is your first traceability tool: the string you place there ends up in the model’s final ID and remains readable everywhere—in your logs, your configuration, your console. Adopt a convention and stick to it.

# Convention: {name}-v{major_version}.{minor_version}
job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-5.6-luna",
    suffix="support-client-v2.1"
)

The number itself must carry information, not just count attempts. A major version changes when the nature of the model changes; a minor version, when you improve without breaking. Concretely, a data fix applied with the same hyperparameters stays on the same major version, while a change of base model requires starting a new one.

Maintaining a model registry

The name alone is not enough: it says nothing about what was in the dataset, the results obtained, or why this version was promoted. A centralized registry keeps these metadata alongside the technical ID and becomes your team’s single source of truth. The class below keeps the essentials in a JSON file: it records a candidate model, promotes one to production while marking the previous one as retired, returns the currently active ID, and displays the history. Take the time to fill in the notes field: a single sentence on what changed since the previous version will save you half a day of archaeology later.

import json
import datetime

class ModelRegistry:
    """Registry of fine-tuned models."""

    def __init__(self, path: str = "model_registry.json"):
        self.path = path
        try:
            with open(path, "r") as f:
                self.registry = json.load(f)
        except FileNotFoundError:
            self.registry = {"models": [], "production": None}

    def record(
        self,
        version: str,
        model_id: str,
        job_id: str,
        dataset_info: dict,
        hyperparams: dict,
        metrics: dict,
        notes: str = ""
    ):
        """Records a new model in the registry."""
        entry = {
            "version": version,
            "model_id": model_id,
            "job_id": job_id,
            "date": datetime.datetime.now().isoformat(),
            "dataset": dataset_info,
            "hyperparameters": hyperparams,
            "metrics": metrics,
            "notes": notes,
            "status": "candidate"
        }
        self.registry["models"].append(entry)
        self._save()
        print(f"Model {version} recorded: {model_id}")

    def promote(self, version: str):
        """Promotes a model to production."""
        old = self.registry["production"]
        for m in self.registry["models"]:
            if m["version"] == version:
                m["status"] = "production"
                self.registry["production"] = version
                break

        if old:
            for m in self.registry["models"]:
                if m["version"] == old:
                    m["status"] = "retired"

        self._save()
        print(f"Production: {old} -> {version}")

    def rollback(self, version: str):
        """Rollback to a previous version."""
        self.promote(version)
        print(f"Rollback performed to {version}")

    def production_model(self) -> str:
        """Returns the ID of the model in production."""
        version = self.registry["production"]
        for m in self.registry["models"]:
            if m["version"] == version:
                return m["model_id"]
        return None

    def history(self):
        """Displays the model history."""
        print(f"{'Version':<12} {'Status':<12} {'Date':<12} {'Val Loss':<10}")
        print("-" * 50)
        for m in self.registry["models"]:
            date = m["date"][:10]
            loss = m["metrics"].get("validation_loss", "N/A")
            print(f"{m['version']:<12} {m['status']:<12} {date:<12} {loss}")

    def _save(self):
        with open(self.path, "w") as f:
            json.dump(self.registry, f, indent=2, ensure_ascii=False)

In practice, the lifecycle of a version comes down to three calls: you record the model coming out of training with its numbers, you promote it when evaluation validates it, and you revert to the previous version if production contradicts the evaluation. This last line should be as trivial to write as the first two.

registry = ModelRegistry()

# Record a new model
registry.record(
    version="v2.1",
    model_id="ft:gpt-5.6-luna:org::support-v2.1:abc123",
    job_id="ftjob-xyz789",
    dataset_info={"train": 350, "validation": 45},
    hyperparams={"n_epochs": 3, "lr": "auto"},
    metrics={"validation_loss": 0.38, "precision_test": 0.93},
    notes="Added 50 fix examples for refunds"
)

# Promote to production
registry.promote("v2.1")

# In case of trouble, rollback
registry.rollback("v2.0")

Rolling back in seconds

A rollback is only valuable if it is fast and requires no thought. Since your application reads the model name from an environment variable, changing versions means changing that variable and updating the registry—no code redeployment, no recompilation. The quick_rollback function goes further by bypassing human choice entirely: it picks up the last available stable version and, if none exists, falls back to the base model. This last branch is your absolute safety net; trigger it voluntarily once in a test environment, while the stakes are zero.

import os

def deploy_model(version: str, registry: ModelRegistry):
    """Deploys a model by updating the environment variable."""
    model_id = None
    for m in registry.registry["models"]:
        if m["version"] == version:
            model_id = m["model_id"]
            break

    if not model_id:
        raise ValueError(f"Version {version} not found")

    # Update configuration
    os.environ["OPENAI_FT_MODEL"] = model_id
    registry.promote(version)

    print(f"Model deployed: {version} ({model_id})")
    return model_id

def quick_rollback(registry: ModelRegistry):
    """Rollback to the last stable version."""
    production = registry.registry["production"]
    versions = [
        m for m in registry.registry["models"]
        if m["version"] != production and m["status"] != "retired"
    ]

    if not versions:
        # Fallback to base model
        os.environ["OPENAI_FT_MODEL"] = "gpt-5.6-luna"
        print("Rollback to base model")
        return

    last_stable = versions[-1]
    deploy_model(last_stable["version"], registry)

Archive the data, not just the models

Versioning a model without versioning what produced it only gets you halfway. The day you want to understand why v2.1 held the tone better than v2.2, you will need the exact three files used to train it, including the test set—without it, no subsequent comparison is fair. Archive them in a folder bearing the version number, at the very moment you record the model.

import shutil

def archive_dataset(version: str, files: list[str], archive_folder: str):
    """Archives training data for a given version."""
    folder = f"{archive_folder}/{version}"
    os.makedirs(folder, exist_ok=True)

    for file in files:
        shutil.copy2(file, folder)

    print(f"Data archived in {folder}")

archive_dataset(
    "v2.1",
    ["train.jsonl", "validation.jsonl", "test.jsonl"],
    "archives/datasets"
)

Automate the full pipeline

These steps, executed by hand, eventually get skipped—especially archiving, which comes last and blocks nothing. Gather them into a single script that chains file upload, job launch, waiting for completion, recording in the registry, and then archiving the data. The wait loop queries the job every thirty seconds and stops on a terminal status; in case of failure, the function returns immediately rather than continuing with a non-existent model.

def deployment_pipeline(
    training_file: str,
    validation_file: str,
    version: str,
    base_model: str = "gpt-5.6-luna",
    hyperparams: dict = None
):
    """Full pipeline: upload, train, evaluate, record."""
    client = OpenAI()
    registry = ModelRegistry()

    # 1. Upload
    print("Uploading data...")
    train = client.files.create(file=open(training_file, "rb"), purpose="fine-tune")
    val = client.files.create(file=open(validation_file, "rb"), purpose="fine-tune")

    # 2. Training
    print("Launching fine-tuning...")
    params = {"training_file": train.id, "validation_file": val.id,
              "model": base_model, "suffix": version}
    if hyperparams:
        params["hyperparameters"] = hyperparams

    job = client.fine_tuning.jobs.create(**params)

    # 3. Wait for completion
    import time
    while True:
        job = client.fine_tuning.jobs.retrieve(job.id)
        if job.status in ("succeeded", "failed"):
            break
        time.sleep(30)

    if job.status == "failed":
        print(f"Failure: {job.error}")
        return None

    # 4. Record
    registry.record(
        version=version,
        model_id=job.fine_tuned_model,
        job_id=job.id,
        dataset_info={"train": training_file, "validation": validation_file},
        hyperparams=hyperparams or {},
        metrics={"trained_tokens": job.trained_tokens}
    )

    # 5. Archive data
    archive_dataset(version, [training_file, validation_file], "archives/datasets")

    print(f"Pipeline complete: {job.fine_tuned_model}")
    return job.fine_tuned_model

Cleaning up

Fine-tuned models are stored on OpenAI and can be deleted when you no longer need them. After a dozen iterations, the list becomes unreadable and manual sorting dangerous. The automatic cleanup below keeps the three most recent versions and never touches the one running in production. Avoid being too aggressive: it is precisely the penultimate version, often deemed useless, that will save you during a rollback.

def clean_old_models(registry: ModelRegistry, keep: int = 3):
    """Deletes old models keeping the last N."""
    models = registry.registry["models"]
    production = registry.registry["production"]

    candidates = [
        m for m in models
        if m["version"] != production
    ]

    to_delete = candidates[:-keep] if len(candidates) > keep else []

    for m in to_delete:
        try:
            client.models.delete(m["model_id"])
            print(f"Deleted: {m['version']} ({m['model_id']})")
        except Exception as e:
            print(f"Error deleting {m['version']}: {e}")

Key takeaways

• Use versioned suffixes to identify each model
• Maintain a centralized registry with metrics and metadata
• Implement a quick rollback mechanism
• Archive training data with each version
• Automate the pipeline and regularly clean up old models: fewer human errors, fewer unnecessary costs

Test your knowledge

Before versioning your models, validate the full fine-tuning cycle.

1. When is fine-tuning the right tool—and when is it not?

Answer: Good for anchoring a style, format, or recurring behavior that prompting does not achieve reliably; bad for injecting fresh knowledge—that is the role of RAG.

2. What makes a training dataset high quality?

Answer: Clean, consistent, and representative examples in conversational JSONL format: a few hundred excellent examples beat thousands of noisy ones.

3. What does DPO bring compared to supervised fine-tuning?

Answer: Preference learning: you provide preferred/rejected response pairs, and the model learns “better” rather than simple imitation—useful for tone and fine trade-offs.

4. How do you validate a fine-tuned model?

Answer: By comparing it to the base model on an evaluation set kept separate from training, then by A/B testing on business metrics—never on impression alone.

5. Why version and plan for rollback?

Answer: Each fine-tuning produces a dated model: in case of regression in production, you switch back to the previous version immediately—the model is managed like code.

Clean data, honest evaluation, controlled versions: fine-tuning is an engineering discipline—this course has given you the full cycle.