Hyperparameters: Epochs, Learning Rate, Batch Size

Update November 2, 2026 — Fine-tuning availability. OpenAI’s self-service fine-tuning is being phased out: since May 7, 2026, organizations that have never run a fine-tuning job can no longer create one; since July 2, you must have used a fine-tuned model within the last 60 days; and starting January 6, 2027, no customer will be able to launch new jobs. Already trained models remain usable until their base model is retired. This course remains useful for understanding and leveraging an existing model; for new projects, prioritize prompt engineering, RAG, and evaluations.

The role of hyperparameters

Your data describes what the model should learn; hyperparameters describe how it learns. By default, OpenAI automatically selects reasonable values based on your dataset size, and for a first job, that’s the right move. But when a model underperforms despite a clean corpus, the difference almost always comes down to these settings, and tweaking them manually can significantly improve results.

The number of epochs

An epoch is one complete pass through your entire training dataset. The more epochs you run, the more times the model sees each example, and the more detail it retains. Too few epochs, and the model hasn’t learned enough: this is underfitting, and your responses still look like those from the base model. Too many epochs, and it memorizes the examples instead of generalizing: this is overfitting, and the model recites your hundred training answers as soon as a question vaguely resembles one, while falling apart on unseen phrasing.

job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-5.6-luna",
    hyperparameters={
        "n_epochs": 3
    }
)

The right setting depends directly on your data volume: the fewer examples you have, the more times you need to pass over them for the signal to stick.

The learning rate multiplier

The learning rate controls the magnitude of adjustments applied to the model’s weights at each training step. Set too high, and the model corrects too aggressively at each step, oscillates around the right answer, and fails to converge; set too low, and it learns very slowly, or not at all, leaving you with a model at the end of training that’s nearly identical to the starting one.

job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-5.6-luna",
    hyperparameters={
        "learning_rate_multiplier": 1.8
    }
)

The default value is calculated automatically by OpenAI based on your dataset size. In practice, the 1.0–2.0 range matches this default behavior and suits most cases. Dropping to 0.5–1.0 makes learning more cautious, which helps on small datasets where a few atypical examples could pull the model too far off course. Going above 2.0 yields aggressive learning, reserved for situations where you’ve observed the model isn’t moving enough—and should be checked immediately against validation loss.

The batch size

Batch size determines how many examples are processed together before updating the model’s weights. A small batch produces frequent updates: learning is noisier, with each example weighing heavily in the correction, but also more responsive. A large batch averages more examples before each correction, resulting in less frequent updates and more stable learning—this is the lever to pull when your curves are jittery for no apparent reason.

job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-5.6-luna",
    hyperparameters={
        "batch_size": 4
    }
)

You don’t have to tune these separately: all three parameters go into the same dictionary.

job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    validation_file=validation_file.id,
    model="gpt-5.6-luna",
    suffix="optimise-v2",
    hyperparameters={
        "n_epochs": 3,
        "learning_rate_multiplier": 1.5,
        "batch_size": 4
    }
)

An optimization method

The classic mistake is changing all three values at once, seeing an improvement, and having no idea which one caused it. Proceed in this order:

1. First run: leave values at default (auto)
2. Analyze the loss: if training loss goes down but validation loss goes up, you’re overfitting
3. Adjust one parameter at a time: never change everything at once
4. Document every run: note the hyperparameters and the results

This log isn’t just paperwork. By the third or fourth run, a table linking a job ID, the hyperparameters used, and the observed outcome beats your memory every time. Here’s how to read the most common symptoms.

The “auto” value

If you don’t specify a hyperparameter, OpenAI automatically chooses an optimal value. So you’re never forced to fill everything in: set what you want to control, and leave the rest to automatic tuning. This is often the best starting point, and it gives you a baseline to compare your later adjustments against.

# Only n_epochs is set; the rest is auto
job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-5.6-luna",
    hyperparameters={
        "n_epochs": 3
        # learning_rate_multiplier and batch_size will be auto
    }
)

Key takeaways

• Always start with default values (auto) before adjusting
• n_epochs: adjust first, based on your data size
• learning_rate_multiplier: adjust if convergence is too slow or too unstable
• batch_size: adjust if training is noisy
• Change only one hyperparameter at a time to understand its impact
• Always use a validation file to detect overfitting