Retry, Exponential Backoff, and Circuit Breaker
On this page
Retry, Exponential Backoff, and Circuit Breaker
Transient errors are normal in production. The OpenAI SDK automatically handles two retries, but for maximum resilience, you should implement your own strategies. The previous lesson taught you to recognize retryable errors; this one teaches you to retry intelligently—without making the situation you’re trying to fix worse.
Exponential Backoff
The principle is simple: each retry waits longer than the previous one. This prevents overloading the API when it’s under pressure. Jitter deserves special attention because its absence causes a classic failure mode: fifty instances of your service receive a 429 at the same second, all wait exactly one second, then all restart together and get rejected again as a group. By adding a random delay, you spread out this restart—that’s the remedy for the thundering herd.
import time
import random
from openai import OpenAI, RateLimitError, InternalServerError, APIConnectionError
client = OpenAI()
def call_with_backoff(prompt: str, max_retries: int = 5) -> str:
"""API call with exponential backoff and jitter."""
for attempt in range(max_retries):
try:
response = client.responses.create(
model="gpt-5.6-terra",
input=prompt
)
return response.output_text
except (RateLimitError, InternalServerError, APIConnectionError) as e:
if attempt == max_retries - 1:
raise # Last attempt, propagate the error
# Exponential backoff: 1s, 2s, 4s, 8s, 16s
base_delay = 2 ** attempt
# Random jitter to avoid thundering herd
jitter = random.uniform(0, base_delay * 0.5)
delay = base_delay + jitter
print(f"Attempt {attempt + 1}/{max_retries} failed. "
f"Retrying in {delay:.1f}s...")
time.sleep(delay)
raise RuntimeError("Should never reach here")
# Usage
result = call_with_backoff("Explain exponential backoff.")
print(result)Notice the final attempt: we propagate the original error instead of swallowing it. The caller thus receives the real diagnostic—429 or 500—and can decide what to do with it.
Using the Tenacity Library
Writing this loop by hand quickly becomes repetitive once you have five functions to protect. For more advanced retry management, tenacity is the go-to tool: the decorator declares the policy, and the function body contains only the business logic call. You get the same ingredients, but named and configurable—the list of exceptions to retry, exponential wait capped at sixty seconds, stopping after five attempts, and automatic logging before each pause.
# pip install tenacity
from tenacity import (
retry,
stop_after_attempt,
wait_exponential,
retry_if_exception_type,
before_sleep_log,
)
from openai import RateLimitError, InternalServerError, APIConnectionError
import logging
logger = logging.getLogger(__name__)
@retry(
retry=retry_if_exception_type(
(RateLimitError, InternalServerError, APIConnectionError)
),
wait=wait_exponential(multiplier=1, min=1, max=60),
stop=stop_after_attempt(5),
before_sleep=before_sleep_log(logger, logging.WARNING),
)
def resilient_call(prompt: str, model: str = "gpt-5.6-terra") -> str:
"""API call with automatic retry via tenacity."""
response = client.responses.create(
model=model,
input=prompt
)
return response.output_text
# Usage — retries are transparent
result = resilient_call("Hello!")
print(result)The Circuit Breaker Pattern
Retry solves incidents lasting a few seconds. It becomes harmful when the outage lasts longer: each user request then consumes five attempts and thirty seconds of waiting before failing, your threads pile up, and it’s your own service that goes down. The circuit breaker prevents your application from continuing to call a failing API.
It works like an electrical circuit breaker, with three states. Closed, everything works normally and calls pass through. Open, the API is considered down and calls are blocked immediately, without even attempting the network—the user receives an error in milliseconds rather than after a timeout. Half-open, once the rest period has elapsed, a test call is sent to check if the API is back: if it succeeds, the circuit closes; if it fails, it reopens for a new cycle.
import time
from enum import Enum
class CircuitState(Enum):
CLOSED = "closed"
OPEN = "open"
HALF_OPEN = "half_open"
class CircuitBreaker:
"""Circuit breaker for API calls."""
def __init__(self, error_threshold: int = 5, reset_timeout: float = 30.0):
self.threshold = error_threshold
self.reset_timeout = reset_timeout
self.consecutive_errors = 0
self.state = CircuitState.CLOSED
self.last_error_time = 0.0
def can_call(self) -> bool:
"""Checks if a call is allowed."""
if self.state == CircuitState.CLOSED:
return True
if self.state == CircuitState.OPEN:
if time.monotonic() - self.last_error_time > self.reset_timeout:
self.state = CircuitState.HALF_OPEN
return True # One test call
return False
# Half-open: only one call allowed
return True
def record_success(self):
"""Records a successful call."""
self.consecutive_errors = 0
self.state = CircuitState.CLOSED
def record_error(self):
"""Records a failure."""
self.consecutive_errors += 1
self.last_error_time = time.monotonic()
if self.consecutive_errors >= self.threshold:
self.state = CircuitState.OPEN
print(f"Circuit OPEN after {self.threshold} consecutive errors")The error counter resets to zero as soon as a call succeeds: it’s consecutive failures that open the circuit, not a historical cumulative count. Client-side integration then comes down to three steps—ask for permission, signal success, signal failure.
# Integration with the OpenAI client
circuit = CircuitBreaker(error_threshold=3, reset_timeout=30.0)
def call_with_circuit_breaker(prompt: str) -> str:
"""API call protected by a circuit breaker."""
if not circuit.can_call():
raise RuntimeError(
f"Circuit open — API unavailable. "
f"Retry in {circuit.reset_timeout}s"
)
try:
response = client.responses.create(
model="gpt-5.6-terra",
input=prompt
)
circuit.record_success()
return response.output_text
except (RateLimitError, InternalServerError, APIConnectionError) as e:
circuit.record_error()
raise
# Usage
try:
result = call_with_circuit_breaker("Hello!")
print(result)
except RuntimeError as e:
print(f"Error: {e}")Combining the Three Patterns
In production, combine backoff, circuit breaker, and timeout. Notice the constructor: the client is created with max_retries=0, which disables the SDK’s internal retries. Without this, your three attempts would multiply by the SDK’s two, and you’d wait much longer than expected. The thirty-second timeout bounds each individual call, the loop handles retries, and the circuit breaker monitors the overall health of the service.
class APIClient:
"""Resilient API client combining all patterns."""
def __init__(self):
self.client = OpenAI(timeout=30.0, max_retries=0)
self.circuit = CircuitBreaker(error_threshold=5, reset_timeout=60.0)
def call(self, prompt: str, max_retries: int = 3) -> str:
if not self.circuit.can_call():
raise RuntimeError("Service temporarily unavailable")
for attempt in range(max_retries):
try:
response = self.client.responses.create(
model="gpt-5.6-terra",
input=prompt
)
self.circuit.record_success()
return response.output_text
except (RateLimitError, InternalServerError) as e:
self.circuit.record_error()
if attempt < max_retries - 1:
delay = (2 ** attempt) + random.uniform(0, 1)
time.sleep(delay)
else:
raise
raise RuntimeError("Failed after all attempts")
# Usage
api = APIClient()
print(api.call("Explain resilience in software engineering."))Before moving on, test this client by simulating an outage: cut the network, launch a dozen calls, and verify that the last ones fail instantly instead of waiting. If that’s not the case, your threshold is too high for the volume being processed.
Key Takeaways
- The SDK handles 2 retries by default — adjust with
max_retries - Exponential backoff with jitter avoids the thundering herd
- The circuit breaker protects your application from prolonged outages
- Use
tenacityfor declarative retry management in production - Combine all three patterns for maximum resilience
- Never retry 400/401/403 errors (client errors)