Rate limits and best practices

Updated November 2, 2026. Based on xAI documentation: the batch discount is 20% on eligible text models, the examples follow the REST API and the official xai-sdk (client.batch.*; xAI’s Batch API does not use OpenAI’s format, so the OpenAI SDK does not apply), and the Responses API takes precedence over Chat Completions, a legacy endpoint at xAI. Sources: Batch API · REST reference · Batch pricing.

Constraints you need to know

The Batch API imposes technical limits that you need to account for in your architecture. Ignoring them leads to rejections, canceled batches, or lost results. This lesson summarizes the constraints documented by xAI and best practices for production use.

Rate limits

Batch creation: the limit of 2 creations per second is rarely a problem, since you generally create one batch per operation.

Adding requests: the limit applies to calls, and each call carries a list of requests. With chunks of 100 requests, it already allows more than 3,000 requests per second.

Request size: the 25 MB limit mainly affects requests loaded with base64-encoded images; a text request rarely exceeds a few kilobytes.

Conversely, batch requests do not count toward the rate limits of the real-time API: a large batch does not slow down your interactive application.

JSONL file limits

If your batch exceeds these limits, split it into several files and create one batch per file; you will aggregate the results afterward.

URL expiration

Signed URLs for image and video results expire after one hour. This is the most critical constraint to handle in production: your retrieval process must download the media immediately, without manual intervention (the telecharger_medias function from lesson 9). For the batch, also monitor expire_time, after which no results are accessible.

Production best practices

Deterministic IDs

Derive your request IDs (batch_request_id, or custom_id in JSONL) from your input data. If you need to resubmit a batch, the same data will produce the same IDs, which makes deduplication easier:

import hashlib


def identifiant_requete(document_id: str, operation: str) -> str:
    """Generates a deterministic request ID."""
    brut = f"{document_id}:{operation}"
    return hashlib.sha256(brut.encode()).hexdigest()[:16]

Handling partial failures

A fully processed batch can still contain failed requests. Always check every result:

reussies, echouees = [], []
jeton = None
while True:
    page = client.batch.list_batch_results(
        batch_id=batch_id, limit=1000, pagination_token=jeton
    )
    reussies.extend(page.succeeded)
    echouees.extend(page.failed)
    if page.pagination_token is None:
        break
    jeton = page.pagination_token

if echouees:
    print(f"{len(echouees)} failed requests to resubmit in a new batch")

Cost tracking

Each batch keeps track of its total cost in ticks (ten-billionths of a dollar):

batch = client.batch.get(batch_id=batch_id)
print("Total cost: %.4f $" % (batch.cost_breakdown.total_cost_usd_ticks / 1e10))

Through the REST API, the cost of each text request appears in its result, in the usage.cost_in_usd_ticks field.

Logs and traceability

Log every operation so you can diagnose problems:

import logging

logger = logging.getLogger("batch_api")

logger.info("Batch created: %s", batch_id)
logger.info("File uploaded: %s, %d requests", file_id, len(documents))
logger.info("Results: %d succeeded, %d failed", len(reussies), len(echouees))

Retry with backoff

For transient errors (timeout, rate limit, server error), retry with an increasing delay:

import time


def appel_avec_retry(fonction, max_essais=3):
    for essai in range(max_essais):
        try:
            return fonction()
        except Exception as e:
            if essai == max_essais - 1:
                raise
            attente = 2 ** essai
            print(f"Retrying in {attente} s: {e}")
            time.sleep(attente)

Pre-production checklist

Before deploying a Batch API pipeline to production, check:

• The model accepts batch processing, and appears in the list of discounted models if cost matters
• Request IDs are unique and deterministic
• The JSONL file is validated before upload (one invalid line cancels the batch)
• Download of multimodal results is automated, within one hour
• Partial failures are detected and resubmitted in a new batch
• Results are retrieved before expire_time
• Logs cover every step of the workflow
• The API key is read from an environment variable, never hardcoded

Key takeaways

• Limits: 2 creations per second, 1,000 request-add calls per 30 seconds, 25 MB per request
• JSONL files: 50,000 requests and 200 MB maximum, one batch per file
• Image and video URLs expire in one hour: download immediately
• Deterministic IDs, checking every result, cost tracking in ticks
• Logs, retry with backoff, and a maximum timeout for production

Test your knowledge

Bulk processing: the Batch API with no gray areas.

1. When is the Batch API the right choice?

Answer: For large volumes without real-time constraints: classification, enrichment, evaluations, bulk generation. Batch requests do not count toward rate limits, and eligible text models get a 20% discount.

2. What is the four-step workflow?

Answer: Create a named batch, add requests to it (through the API, or by creating the batch from an uploaded JSONL file), track its counters, then retrieve the results, as they come in or at the end. Processing starts on its own, with no launch step.

3. What can you put in a batch?

Answer: Responses requests (recommended) and Chat Completions (legacy), with server tools and function calling, as well as image and video generations and edits, and video extensions. Only text tokens for eligible models get the discount.

4. How do you track execution?

Answer: Through the batch counters (num_pending, num_success, num_error…), the status of each request (pending, succeeded, failed, cancelled), and the paginated results, which separate successes from failures. The batch is fully processed when num_pending is zero.

5. What best practices apply to the limits?

Answer: Add requests in chunks, split large files into one batch per file, download media within one hour, retrieve results before expire_time, and resubmit failures in a new batch: design the batch as a robust pipeline, not as a blind bulk submission.

Well-formed batches, tracked counters, retried failures: the Batch API turns volume into routine, at the best cost.