Deploy a fine-tuned model — Fine-Tuning OpenAI Models

Lesson updated November 3, 2026 based on OpenAI documentation: temperature and top_p are only accepted with a reasoning effort of none, whereas GPT-5.6 models reason by default (medium). Calls that set these parameters must therefore specify reasoning={"effort": "none"} (API Responses) or reasoning_effort="none" (Chat Completions). Sources: Using GPT-6 · GPT-5.6 Terra.

November 2, 2026 update — Fine-tuning availability. OpenAI’s self-service fine-tuning is being phased out: since May 7, 2026, organizations that have never performed fine-tuning can no longer create jobs; since July 2, you must have used a fine-tuned model within the last 60 days; and starting January 6, 2027, no customer will be able to launch new jobs. Already trained models remain usable until their base model is retired. This course remains useful for understanding and leveraging an existing model; for new projects, prioritize prompt engineering, RAG, and evaluations.

From development to production

Your fine-tuned model is evaluated and approved. It’s time to deploy it to production. This lesson covers best practices for a reliable, secure, and maintainable deployment—everything that separates a model running on your laptop from a service your users can hit continuously without you being on call.

Let’s start with the basics: a fine-tuned model works exactly like any other OpenAI model. The only difference is the model name, which becomes a long identifier including your organization, your suffix, and a job ID. No extra parameters, no special endpoints.

from openai import OpenAI

client = OpenAI()

# The fine-tuned model has a unique identifier
MODELE_PRODUCTION = "ft:gpt-5.6-luna:votre-org::mon-assistant-v3:abc12345"

response = client.responses.create(
    model=MODELE_PRODUCTION,
    input="Hello, I have an issue with my order #12345."
)

print(response.output_text)

Centralize configuration

This identifier has one quirk that should govern your entire architecture: it changes with every new version of your model. Hard-coding it throughout your application forces a full redeployment for the slightest update, and a tedious search the day it appears in three different places. Put it in an environment variable, with the base model as a fallback value: your application will start even if the variable isn’t set, avoiding a silent service incident on a Sunday night.

import os

class Config:
    """Centralized model configuration."""

    # Environment variable for the production model
    MODEL_ID = os.environ.get(
        "OPENAI_FT_MODEL",
        "gpt-5.6-luna"  # fallback to base model
    )

    # System prompt (can be empty if integrated into fine-tuning)
    SYSTEM_PROMPT = os.environ.get("SYSTEM_PROMPT", "")

    # Generation parameters
    MAX_TOKENS = int(os.environ.get("MAX_TOKENS", "1024"))
    TEMPERATURE = float(os.environ.get("TEMPERATURE", "0.7"))

The same logic applies to the calls themselves. Instead of scattering client.responses.create calls throughout your business logic, encapsulate them in a single service. You gain a mandatory choke point where you can later plug in logging, metrics, and fallback without reopening twenty files. Notice how the service below composes an optional context with the question, and how it catches any exception to redirect it to the base model.

from openai import OpenAI

class AssistantService:
    """Service encapsulating calls to the fine-tuned model."""

    def __init__(self):
        self.client = OpenAI()
        self.model = Config.MODEL_ID

    def repondre(self, question: str, contexte: str = "") -> str:
        """Generates a response."""
        input_msg = question
        if contexte:
            input_msg = f"Context: {contexte}\n\nQuestion: {question}"

        try:
            response = self.client.responses.create(
                model=self.model,
                reasoning={"effort": "none"},  # temperature requires effort none
                input=input_msg,
                max_output_tokens=Config.MAX_TOKENS,
                temperature=Config.TEMPERATURE
            )
            return response.output_text

        except Exception as e:
            # Fallback to base model on error
            return self._fallback(input_msg, str(e))

    def _fallback(self, input_msg: str, erreur: str) -> str:
        """Fallback to base model."""
        print(f"Fallback activated: {erreur}")
        response = self.client.responses.create(
            model="gpt-5.6-luna",
            instructions=Config.SYSTEM_PROMPT,
            input=input_msg
        )
        return response.output_text

Resist incidents

A degraded response is better than no response. If your fine-tuned model is temporarily unavailable, it’s better to switch to the base model—with the system prompt you kept precisely for this case—than to display an error to the user. The following function chains two complementary mechanisms: first, three attempts on the primary model, spaced by a delay that doubles with each failure to avoid worsening a temporary overload; then, only if everything fails, a call to the fallback model. Keep the fallback flag returned in the result, as it is the only usable trace of how often this mechanism actually triggers.

import time

def appel_resilient(
    client: OpenAI,
    modele_principal: str,
    modele_fallback: str,
    input_msg: str,
    max_retries: int = 3
) -> dict:
    """Resilient call with retry and fallback."""

    for tentative in range(max_retries):
        try:
            response = client.responses.create(
                model=modele_principal,
                input=input_msg,
                timeout=30
            )
            return {
                "reponse": response.output_text,
                "modele_utilise": modele_principal,
                "fallback": False
            }
        except Exception as e:
            print(f"Attempt {tentative + 1} failed: {e}")
            if tentative < max_retries - 1:
                time.sleep(2 ** tentative)  # exponential backoff

    # Fallback
    try:
        response = client.responses.create(
            model=modele_fallback,
            input=input_msg
        )
        return {
            "reponse": response.output_text,
            "modele_utilise": modele_fallback,
            "fallback": True
        }
    except Exception as e:
        return {"erreur": str(e), "fallback": True}

Rate limiting deserves separate treatment because it isn’t an outage but a regulation: fine-tuned models share the same rate limits as base models. A RateLimitError simply asks you to wait, and the progressive wait below is enough to absorb a spike. If the loop hits its ceiling, however, the problem is no longer conjunctural and you need to review your quota.

from openai import RateLimitError
import time

def appel_avec_rate_limit(client, modele, input_msg, max_wait=60):
    """Handles rate limiting with progressive wait."""
    wait = 1
    while wait <= max_wait:
        try:
            return client.responses.create(
                model=modele,
                input=input_msg
            )
        except RateLimitError:
            print(f"Rate limit reached, waiting {wait}s...")
            time.sleep(wait)
            wait *= 2

    raise Exception("Persistent rate limit — check your quotas")

Log what matters

Structured logging turns opaque incidents into quick diagnostics. For each call, record the model actually used, input and output volume, latency, and the fallback flag. This information is enough to answer the questions that arise during an incident: how long has latency been drifting, is the cost increase due to traffic or longer responses, and how many users were actually served by the base model.

import logging
import datetime

logger = logging.getLogger("fine_tuning_prod")

def log_appel(
    modele: str,
    input_msg: str,
    output: str,
    latence_ms: float,
    fallback: bool
):
    """Structured log for each production call."""
    logger.info(
        "appel_modele",
        extra={
            "modele": modele,
            "input_tokens": len(input_msg.split()),
            "output_tokens": len(output.split()),
            "latence_ms": latence_ms,
            "fallback": fallback,
            "timestamp": datetime.datetime.now().isoformat()
        }
    )

On the configuration side, everything goes into an environment file that must never enter your repository: it contains your API key, and a published key is a compromised key.

# .env (do not commit)
OPENAI_API_KEY=sk-...
OPENAI_FT_MODEL=ft:gpt-5.6-luna:org::suffix:abc12345
OPENAI_FT_MODEL_FALLBACK=gpt-5.6-luna
SYSTEM_PROMPT="You are a professional customer support assistant."
MAX_TOKENS=1024
TEMPERATURE=0.7

Before opening the floodgates

Before exposing the service to your users, go through the checks in this lesson one by one. The model must have been evaluated and approved against the threshold you defined in advance; its name must live in an environment variable and nowhere hard-coded; a fallback mechanism must be in place. These three points condition everything else, because without them you are putting online a model whose measured quality, exact address, and behavior when the API responds poorly are unknown.

Next, check instrumentation. Logging must be enabled and record at minimum the model used, latency, and tokens; rate limiting must be handled rather than suffered; alerts must be configured on error rate and P95 latency, otherwise your first unhappy user will play the role of the probe. Finally, rollback must be documented and testable—a rollback never actually executed remains an intention, not a mechanism.

Key takeaways

Use environment variables for the model name
Always implement a fallback to the base model
Log every call with the model used and latency
Handle rate limiting with exponential backoff
Test your fallback mechanism before deploying