Exponential Backoff on HTTP 429

Understanding the 429 Error

An HTTP 429 error means you’ve exceeded the API’s rate limits. The xAI server temporarily refuses your requests to protect the infrastructure. This error is normal in a batch processing context: it simply indicates you’re sending requests too quickly.

The appropriate reaction isn’t to give up, but to wait before retrying. Exponential backoff is the standard strategy for handling this situation.

How Exponential Backoff Works

Instead of retrying immediately (which would trigger another 429), you wait an increasing delay between each attempt:

• 1st failure: wait 1 second
• 2nd failure: wait 2 seconds
• 3rd failure: wait 4 seconds
• 4th failure: wait 8 seconds
• 5th failure: give up

The delay doubles with each attempt, hence the term “exponential.” This gives the server time to recover capacity.

Manual Implementation

import asyncio
import random
from openai import RateLimitError

async def request_with_backoff(
    prompt: str,
    max_retries: int = 5,
    base_delay: float = 1.0
) -> str:
    for attempt in range(max_retries):
        try:
            response = await client.chat.completions.create(
                model="grok-4.5",
                messages=[{"role": "user", "content": prompt}]
            )
            return response.choices[0].message.content

        except RateLimitError:
            if attempt == max_retries - 1:
                raise  # last attempt, propagate the error

            delay = base_delay * (2 ** attempt)
            jitter = random.uniform(0, delay * 0.1)
            wait_time = delay + jitter

            print(f"Rate limit hit, attempt {attempt + 1}/{max_retries}. "
                  f"Waiting {wait_time:.1f}s")
            await asyncio.sleep(wait_time)

The jitter (random variation) is important. If multiple coroutines receive a 429 at the same time, without jitter they would all retry at exactly the same moment, triggering another 429. Jitter desynchronizes the attempts.

Using the tenacity Library

For production use, the tenacity library simplifies implementation:

from tenacity import (
    retry,
    stop_after_attempt,
    wait_exponential,
    retry_if_exception_type
)
from openai import RateLimitError

@retry(
    stop=stop_after_attempt(5),
    wait=wait_exponential(multiplier=1, min=1, max=60),
    retry=retry_if_exception_type(RateLimitError)
)
async def robust_request(prompt: str) -> str:
    response = await client.chat.completions.create(
        model="grok-4.5",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

The @retry decorator intercepts RateLimitError, waits an exponential delay (between 1 and 60 seconds), and retries up to 5 times. The function code remains clean and readable.

Combining Backoff and Semaphore

In practice, you combine backoff with a semaphore for full control:

async def resilient_batch(prompts: list[str], max_concurrent: int = 5) -> list[str]:
    sem = Semaphore(max_concurrent)

    async def single_request(prompt: str) -> str:
        async with sem:
            return await request_with_backoff(prompt)

    tasks = [single_request(p) for p in prompts]
    return await asyncio.gather(*tasks, return_exceptions=True)

The semaphore prevents overloading the API, and backoff handles residual 429 errors. This double protection is the recommended configuration for production.

Adapting Backoff to Error Type

Not all errors justify a retry. Here is a classification:

from openai import RateLimitError, APITimeoutError, APIConnectionError, BadRequestError

async def smart_retry(prompt: str, max_retries: int = 5) -> str:
    for attempt in range(max_retries):
        try:
            return await client.chat.completions.create(
                model="grok-4.5",
                messages=[{"role": "user", "content": prompt}]
            )
        except RateLimitError:
            delay = 2 ** attempt
            await asyncio.sleep(delay)
        except (APITimeoutError, APIConnectionError):
            delay = 2 ** attempt
            await asyncio.sleep(delay)
        except BadRequestError:
            raise  # error in prompt, retry useless

429 errors (rate limit), timeouts, and connection errors deserve a retry. However, a BadRequestError (invalid prompt) won’t be resolved by retrying.

Key Takeaways

• A 429 error is normal and expected during batch processing
• Exponential backoff doubles the delay between each attempt (1s, 2s, 4s, 8s…)
• Jitter desynchronizes new attempts to avoid collisions
• The tenacity library simplifies implementation in production
• Always combine semaphore and backoff for maximum resilience
• Do not retry invalid request errors (BadRequestError)