Inline Batching: Up to 10K Requests Without a File

Updated November 3, 2026: November 3 update: files.download returns an HTTP response (httpx2.Response); read its content with .read() before calling .decode().

The Lightweight Alternative to File Batching

Inline batching lets you submit batch requests directly in the API call, skipping the file upload step. This is the ideal method when you have a moderate volume of requests (up to 10,000) and want to simplify your pipeline.

File Batching vs. Inline Batching

Before diving into the code, let’s clarify the differences between the two approaches:

Creating an Inline Job

Instead of passing input_files, you pass a requests parameter containing your list of requests directly:

from mistralai.client import Mistral

client = Mistral(api_key="your-api-key")

# Prepare inline requests
inline_requests = [
    {
        "custom_id": f"req-{i:04d}",
        "body": {
            "max_tokens": 150,
            "messages": [
                {
                    "role": "system",
                    "content": "You are a text classification expert."
                },
                {
                    "role": "user",
                    "content": f"Classify this text: {text}"
                }
            ]
        }
    }
    for i, text in enumerate(my_texts)
]

# Create the job with inline requests
job = client.batch.jobs.create(
    requests=inline_requests,
    model="mistral-small-latest",
    endpoint="/v1/chat/completions"
)

print(f"Inline job created: {job.id}")
print(f"Number of requests: {len(inline_requests)}")

The code difference is minimal: requests= instead of input_files=. The rest of the workflow (status tracking, downloading results) is identical.

Worked Example: Ticket Classification

Here is a complete example of classifying customer support tickets:

from mistralai.client import Mistral
import json
import time

client = Mistral(api_key="your-api-key")

# Simulate support tickets
tickets = [
    {"id": "TK-001", "message": "I cannot log in to my account"},
    {"id": "TK-002", "message": "How do I cancel my subscription?"},
    {"id": "TK-003", "message": "The app crashes on startup"},
    {"id": "TK-004", "message": "Can you add dark mode?"},
    {"id": "TK-005", "message": "My invoice is incorrect"},
]

# Build inline requests
requests = [
    {
        "custom_id": ticket["id"],
        "body": {
            "max_tokens": 50,
            "messages": [
                {
                    "role": "system",
                    "content": (
                        "Classify the ticket into one of these categories: "
                        "AUTHENTICATION, BILLING, BUG, "
                        "FEATURE, SUBSCRIPTION. "
                        "Reply with the category only."
                    )
                },
                {"role": "user", "content": ticket["message"]}
            ]
        }
    }
    for ticket in tickets
]

# Launch the inline batch
job = client.batch.jobs.create(
    requests=requests,
    model="mistral-small-latest",
    endpoint="/v1/chat/completions"
)

# Wait and retrieve results
while True:
    status = client.batch.jobs.get(job_id=job.id)
    if status.status in ("SUCCESS", "FAILED", "CANCELLED"):
        break
    time.sleep(5)

# Parse results
if status.status == "SUCCESS":
    data = client.files.download(file_id=status.output_file)
    for line in data.read().decode("utf-8").strip().split("\n"):
        entry = json.loads(line)
        ticket_id = entry["custom_id"]
        category = entry["response"]["body"]["choices"][0]["message"]["content"]
        print(f"{ticket_id} -> {category.strip()}")

When to Choose Each Approach

The deciding factor isn’t just volume; it’s where batching fits in your workflow. During prototyping, you test a prompt on a few hundred examples, refine it, and rerun. Writing an intermediate JSONL file at every iteration only slows down the loop. Inline batching removes this friction—your script generates requests on the fly, submits them in the same call, and you iterate in minutes. The same logic applies to lightweight integrations, one-off analysis scripts, or daily processing that stays under 10,000 requests.

File batching takes the lead once batching becomes a production link. First, mechanically: beyond 10,000 requests, it’s the only option. But more importantly, structurally: an uploaded file is a durable artifact. You can reuse it for multiple jobs, keep it as a record of exactly what was submitted, and cleanly separate the preparation step (which builds, validates, and versions the file) from the execution step (which launches the job). When a batch job runs nightly on customer data, this traceability isn’t a luxury; it’s what lets you answer the question, “What did we send to the model on March 12?” six months later.

In practice, many teams use both: inline during prompt development, then switch to file batching when industrializing. Since the code for tracking and retrieving results remains identical, migration is limited to replacing requests= with input_files=.

Key Takeaways

• Inline batching removes the upload step by passing requests directly via the requests parameter
• Limit of 10,000 requests per inline job, versus 1 million for file batching
• Status tracking and result downloading are identical for both modes
• Prefer inline for prototyping and moderate volumes; use file batching for large-scale production