Production Best Practices

Lesson updated November 3, 2026 based on OpenAI documentation: temperature and top_p are only accepted with a reasoning effort of none, whereas GPT-5.6 models reason by default (medium). Calls that set these parameters must therefore specify reasoning={"effort": "none"} (Responses API) or reasoning_effort="none" (Chat Completions). Sources: Using GPT-6 · GPT-5.6 Terra.

This lesson covers the essential best practices for deploying an OpenAI API-based application to production. Think of this as your pre-launch checklist. Previous lessons gave you the building blocks—error handling, rate limits, resilience, observability, and costs. Now it’s time to assemble them into an application that a colleague can take over and run without you.

1. Client Architecture

Singleton with Centralized Configuration

Instantiating an OpenAI client on every call wastes the underlying HTTP connection and, more seriously, scatters your configuration. You end up with a 30-second timeout here and the default there, with no way to know which one applies in production. The lru_cache decorator guarantees a single instance, configured in one place.

from openai import OpenAI
from functools import lru_cache

@lru_cache(maxsize=1)
def get_client() -> OpenAI:
    """Singleton client with production configuration."""
    return OpenAI(
        max_retries=3,
        timeout=30.0,
    )

# Everywhere in your code:
client = get_client()

Environment Variables

The same principle applies to settings that vary between environments. Check for required variables at startup: it’s better for a service to refuse to start with a clear message than to start and fail on the first user call. This makes the model, token limit, and temperature adjustable without redeployment.

import os

# REQUIRED: never hard-code keys
REQUIRED_ENV = ["OPENAI_API_KEY"]

for var in REQUIRED_ENV:
    if not os.environ.get(var):
        raise RuntimeError(f"Missing variable {var}")

# Environment-specific configuration
MODEL = os.environ.get("OPENAI_MODEL", "gpt-5.6-terra")
MAX_TOKENS = int(os.environ.get("OPENAI_MAX_TOKENS", "4096"))
TEMPERATURE = float(os.environ.get("OPENAI_TEMPERATURE", "0.7"))

2. Prompt Management

Separate Prompts from Code

A prompt is a business asset, just like a business rule. Burying it in an f-string inside a function makes it impossible for someone who knows the business but not Python to review. Isolate it as a parameterized template: the function then reduces to formatting and calling, and a temperature of 0.0 makes classification reproducible.

# prompts/classification.txt
PROMPT_CLASSIFICATION = """You are a text classifier.
Classify the following text into one of these categories:
{categories}

Text: {texte}

Reply only with the category name."""

def classifier(texte: str, categories: list[str]) -> str:
    prompt = PROMPT_CLASSIFICATION.format(
        categories=", ".join(categories),
        texte=texte
    )
    response = get_client().responses.create(
        model=MODEL,
        reasoning={"effort": "none"},  # temperature requires effort none
        input=prompt,
        temperature=0.0
    )
    return response.output_text.strip()

Version Your Prompts

Changing a prompt changes system behavior just as surely as changing code, but without any unit test noticing. Keep successive versions and select the active one via configuration. This lets you compare two formulations on real traffic and revert to the previous one with an environment variable if the new one degrades results.

PROMPTS = {
    "classification_v1": "Classify: {texte}",
    "classification_v2": "Analyze and classify precisely: {texte}",
}

# In production, use a specific version
ACTIVE_PROMPT = os.environ.get("PROMPT_VERSION", "classification_v2")

3. Timeouts and Limits

Bound any input from outside before it reaches the API. A user pasting a 200-page PDF into your input field must not be able to trigger a multi-dollar call or cause the request to fail due to context overflow. Preventive truncation, combined with the output cap and temperature from configuration, makes the cost and latency of each call predictable.

def appel_avec_garde_fous(prompt: str) -> str:
    """API call with all safeguards."""
    # Limit prompt size
    MAX_PROMPT_CHARS = 50000
    if len(prompt) > MAX_PROMPT_CHARS:
        prompt = prompt[:MAX_PROMPT_CHARS]
        print(f"Prompt truncated to {MAX_PROMPT_CHARS} characters")

    response = get_client().responses.create(
        model=MODEL,
        reasoning={"effort": "none"},  # temperature requires effort none
        input=prompt,
        max_output_tokens=MAX_TOKENS,
        temperature=TEMPERATURE,
    )

    return response.output_text

4. Model Fallback

When a model is saturated or temporarily unavailable, returning an error isn’t your only option. Switching to a secondary model degrades quality but preserves the service—a trade-off often preferable for a support assistant, less so for contract analysis. The loop iterates through models in order of preference and raises an exception only if all fail.

from openai import RateLimitError, InternalServerError

def appel_avec_fallback(prompt: str) -> str:
    """Tries Terra then falls back to Luna on error."""
    modeles = ["gpt-5.6-terra", "gpt-5.6-luna"]

    for modele in modeles:
        try:
            response = get_client().responses.create(
                model=modele,
                input=prompt
            )
            return response.output_text
        except (RateLimitError, InternalServerError) as e:
            print(f"{modele} unavailable: {e}")
            continue

    raise RuntimeError("All models are unavailable")

5. Batch Processing

To process a thousand documents, sequential calls are too slow and simultaneous calls blow up your rate limits. A semaphore strikes the right balance. Two details make this implementation robust: return_exceptions=True prevents a single failure from canceling the whole batch, and sorting by index restores the original order, which asyncio.gather does not guarantee in execution time.

import asyncio
from openai import AsyncOpenAI

async def traiter_batch(prompts: list[str], max_concurrent: int = 5) -> list:
    """Processes a batch of prompts with limited concurrency."""
    client = AsyncOpenAI()
    semaphore = asyncio.Semaphore(max_concurrent)
    resultats = []

    async def traiter_un(index: int, prompt: str):
        async with semaphore:
            response = await client.responses.create(
                model="gpt-5.6-terra",
                input=prompt
            )
            return index, response.output_text

    tasks = [traiter_un(i, p) for i, p in enumerate(prompts)]
    results = await asyncio.gather(*tasks, return_exceptions=True)

    # Sort by index to maintain order
    sorted_results = sorted(
        [(idx, res) for idx, res in results if not isinstance(res, Exception)],
        key=lambda x: x[0]
    )
    return [res for _, res in sorted_results]

# prompts = ["Question 1", "Question 2", "Question 3"]
# résultats = asyncio.run(traiter_batch(prompts))

6. Testing and Validation

You need two different types of tests. The first checks the plumbing—the key works, the network passes, the model responds—and fits well in a startup health check. The second, more original, tests the prompt quality itself: you submit a case where you know the correct answer and verify it appears. These assertions are your safety net against regressions when you rephrase a template.

def test_api_disponible():
    """Verifies that the API is accessible."""
    try:
        response = get_client().responses.create(
            model="gpt-5.6-terra",
            input="test",
            max_output_tokens=5
        )
        assert response.output_text is not None
        print("API OK")
        return True
    except Exception as e:
        print(f"API KO: {e}")
        return False

def test_prompt_quality(prompt: str, attendu: str) -> bool:
    """Verifies that a prompt returns a consistent result."""
    response = get_client().responses.create(
        model="gpt-5.6-terra",
        reasoning={"effort": "none"},  # temperature requires effort none
        input=prompt,
        temperature=0.0
    )
    resultat = response.output_text.lower()
    return attendu.lower() in resultat

# Prompt regression tests
assert test_prompt_quality(
    "Classify 'excellent product': positive or negative?",
    "positive"
)

Build a dozen test cases like this, representative of your real use cases, and run them every time you modify a prompt. A ten-minute test suite will save you weeks of doubt about perceived quality.

7. Production Deployment Checklist

Before deploying, verify each point:

Infrastructure

Environment variables configured (OPENAI_API_KEY, etc.)
Timeouts and max_retries configured
Structured logging in place
Cost monitoring enabled

Resilience

Complete error handling (all HTTP codes)
Exponential backoff for retries
Circuit breaker for prolonged outages
Fallback to an alternative model

Security

API key never exposed in code or logs
User input validation
Client-side rate limiting
No sensitive data in prompts

Performance

Streaming enabled for user interfaces
Batch processing for high-volume tasks
Caching for repetitive requests
Model choice adapted to each task

Costs

Daily budget defined
Alerts configured
max_output_tokens set where possible
Most cost-effective model selected per task

Key Takeaways

Centralize client configuration with a singleton
Separate and version your prompts outside code
Implement model fallback for high availability
Use batch processing with limited concurrency for volume
Test your prompts with quality assertions
Follow the complete checklist before every production deployment