Training Monitoring

Update November 2, 2026 — Fine-tuning availability. OpenAI’s self-service fine-tuning is being phased out: since May 7, 2026, organizations that have never used it can no longer create jobs; since July 2, you must have used a fine-tuned model within the last 60 days; and starting January 6, 2027, no customer will be able to launch new jobs. Already trained models remain usable until their base model is retired. This course remains useful for understanding and leveraging an existing model; for new projects, prioritize prompt engineering, RAG, and evaluations.

Why Monitor Training

Launching a fine-tuning job and forgetting it leaves you with a fait accompli: an hour later, you discover a model without knowing what happened in between. By observing metrics while training runs, you catch problems early, decide to adjust hyperparameters before burning your entire budget, and above all understand why one run outperformed another. Monitoring turns a series of trials into an engineering discipline.

The Two Metrics to Read

Training loss measures how wrong the model is on the training data—the very data it is currently ingesting. It should decrease over steps: that’s the sign learning is happening. A loss that stalls or climbs back up indicates a problem, usually a poorly calibrated learning rate or a corpus that’s too heterogeneous.

Validation loss measures performance on data the model has never seen during training—the file you set aside during the split. This is the most reliable metric for evaluating your model’s actual quality, because it’s the only one the model can’t improve by memorizing. It’s their joint evolution that provides the diagnosis.

The second row is the one that costs the most in production: a training loss that keeps decreasing while validation loss climbs back up describes a model memorizing your examples by heart. On paper, everything seems to be improving; in reality, each additional step degrades its ability to answer a question it has never seen.

Tracking Events via the API

The API doesn’t push notifications: you must poll the job at regular intervals. The loop below fetches new events, displays them in chronological order while avoiding re-displaying those already seen, and returns control as soon as the job reaches a terminal state.

from openai import OpenAI
import time

client = OpenAI()

def monitor_job(job_id: str):
    """Monitors a job in real time."""
    last_event = None

    while True:
        job = client.fine_tuning.jobs.retrieve(job_id)

        # Fetch new events
        events = client.fine_tuning.jobs.list_events(
            fine_tuning_job_id=job_id,
            limit=50
        )

        for event in reversed(events.data):
            if last_event and event.id == last_event:
                continue
            print(f"[{event.created_at}] {event.message}")

        if events.data:
            last_event = events.data[0].id

        if job.status in ("succeeded", "failed", "cancelled"):
            print(f"\nJob finished: {job.status}")
            if job.status == "succeeded":
                print(f"Model: {job.fine_tuned_model}")
            return job

        time.sleep(30)

# Usage
job = monitor_job("ftjob-xxxxxxxxxxxxxxxx")

Events don’t just contain text messages: they also carry loss values at each step. Extracting them lets you plot a curve rather than read log lines, and it’s on a curve that you spot a validation loss spike at a glance.

def extract_metrics(job_id: str) -> dict:
    """Extracts loss metrics from a completed job."""
    events = client.fine_tuning.jobs.list_events(
        fine_tuning_job_id=job_id,
        limit=200
    )

    training_losses = []
    validation_losses = []

    for event in events.data:
        data = event.data
        if data and "train_loss" in str(data):
            training_losses.append(data)
        if data and "valid_loss" in str(data):
            validation_losses.append(data)

    return {
        "training": training_losses,
        "validation": validation_losses
    }

Checkpoints and Intermediate Results

OpenAI saves checkpoints during training. If the job fails or you cancel it, you can sometimes recover a usable intermediate checkpoint—in other words, an interrupted run isn’t necessarily a lost run. This is also a valuable recourse in case of late overfitting: the checkpoint taken before the validation loss spike may be better than the final model.

# List checkpoints for a job
checkpoints = client.fine_tuning.jobs.checkpoints.list(
    fine_tuning_job_id=job.id
)

for cp in checkpoints.data:
    print(f"Step {cp.step_number}")
    print(f"  Model: {cp.fine_tuned_model_checkpoint}")
    if cp.metrics:
        print(f"  Training loss: {cp.metrics.train_loss:.4f}")
        if hasattr(cp.metrics, "valid_loss"):
            print(f"  Validation loss: {cp.metrics.valid_loss:.4f}")

Knowing When to Stop

Three signals justify interrupting a training run in progress. The first is a validation loss that has been climbing for several steps, not just one: an isolated value is noise, a trend over five points is overfitting. The second is a training loss close to zero, meaning the model is reproducing your examples verbatim and has nothing left to learn from them. The third is more prosaic: your tests aren’t improving from one checkpoint to the next, and each additional step just adds to the bill.

# If you detect a problem, cancel the job
client.fine_tuning.jobs.cancel(job.id)
print("Job cancelled. Adjust your data or hyperparameters and relaunch.")

A Summary Report per Run

Finish each training session by producing an actionable record. The following function gathers into one output the job ID, base model, epochs used, number of trained tokens, and metrics from the last checkpoint. Paste these reports into your run log: in a month, they’ll tell you why you kept this version rather than another.

def job_report(job_id: str):
    """Generates a summary report for the job."""
    job = client.fine_tuning.jobs.retrieve(job_id)

    print("=" * 50)
    print(f"Job: {job.id}")
    print(f"Base model: {job.model}")
    print(f"Status: {job.status}")
    print(f"Epochs: {job.hyperparameters.n_epochs}")

    if job.status == "succeeded":
        print(f"Fine-tuned model: {job.fine_tuned_model}")
        print(f"Trained tokens: {job.trained_tokens}")

    if job.status == "failed":
        print(f"Error: {job.error}")

    # Checkpoints
    checkpoints = client.fine_tuning.jobs.checkpoints.list(
        fine_tuning_job_id=job_id
    )
    if checkpoints.data:
        latest = checkpoints.data[0]
        print(f"Latest checkpoint: step {latest.step_number}")
        if latest.metrics:
            print(f"  Train loss: {latest.metrics.train_loss:.4f}")

    print("=" * 50)

job_report("ftjob-xxxxxxxxxxxxxxxx")

Key Takeaways

• Always monitor validation loss—it’s your true quality indicator
• If validation loss climbs, you’re overfitting: reduce epochs
• Checkpoints let you recover an intermediate state
• Automate monitoring for long-running jobs
• Document metrics from each run to compare your iterations