Using Semaphores to Limit Concurrent Requests

The problem with uncontrolled concurrency

Firing off all your requests in parallel without limits is a common mistake. If you have 200 questions to process and launch them all simultaneously, you’ll hit API rate limits, receive HTTP 429 errors, and potentially get your API key temporarily blocked.

A semaphore is the standard solution for controlling the number of concurrent requests. It’s a counter that allows a maximum number of coroutines to run in parallel.

How semaphores work

asyncio.Semaphore acts as a counter lock. When a coroutine enters the async with semaphore block, the counter decreases. When it exits, the counter increases. If the counter hits zero, subsequent coroutines wait until a slot opens up.

import asyncio
from asyncio import Semaphore

semaphore = Semaphore(5)  # maximum 5 simultaneous requests

async def process_request(request: str):
    async with semaphore:
        # This section is limited to 5 parallel executions
        chat = client.chat.create(
            model="grok-4.20-0309-reasoning",
            max_tokens=100
        )
        chat.append(user(request))
        return await chat.sample()

In this example, even if you launch 100 coroutines with asyncio.gather, only 5 will run simultaneously. The other 95 will wait their turn.

Complete pattern with the OpenAI SDK

Here’s the same pattern using AsyncOpenAI, which is often preferred for compatibility:

from openai import AsyncOpenAI
import httpx

client = AsyncOpenAI(
    api_key=os.getenv("XAI_API_KEY"),
    base_url="https://api.x.ai/v1",
    timeout=httpx.Timeout(3600.0)
)

async def send_request(sem: Semaphore, request: str) -> dict:
    async with sem:
        return await client.chat.completions.create(
            model="grok-4.20-0309-reasoning",
            messages=[{"role": "user", "content": request}]
        )

async def process_all(requests: list[str], max_concurrent: int = 5):
    sem = Semaphore(max_concurrent)
    tasks = [send_request(sem, r) for r in requests]
    return await asyncio.gather(*tasks)

The max_concurrent parameter is configurable. Set it based on your rate limits (lesson 7).

Choosing the right semaphore value

Calculate the optimal value from your rate limits, but not directly: you factor in request duration. If your tier allows 60 RPM and each request takes about 10 seconds, a semaphore of 5 produces at most 5 simultaneous requests, or about 30 requests per minute—comfortably under the limit. Apply this reasoning (RPM limit × average duration ÷ 60) to your own case. It explains why reasoning models require a lower semaphore: their requests take longer, so at equal concurrency, throughput per minute drops and the risk of momentary saturation increases. When in doubt, the empirical approach is safest: start low, between 3 and 5, observe the 429 error rate, and gradually increase until you find the balance between throughput and stability.

Approximate calculation

Requests per minute = semaphore * (60 / average_duration_seconds)

Example: semaphore=5, duration=10s → 5 * (60/10) = 30 RPM
Example: semaphore=10, duration=5s → 10 * (60/5) = 120 RPM

Common mistake: global vs. local semaphore

# BAD: new semaphore on every call
async def bad_request(request: str):
    sem = Semaphore(5)  # each call creates its own semaphore
    async with sem:
        return await client.chat.completions.create(...)

# GOOD: semaphore shared across all calls
sem = Semaphore(5)

async def good_request(request: str):
    async with sem:
        return await client.chat.completions.create(...)

The semaphore must be shared across all coroutines. If you create it locally inside each function, every coroutine will have its own semaphore and the limit won’t work.

Key takeaways

• A semaphore limits the number of simultaneous API requests to respect rate limits
• Use async with semaphore to protect API calls
• Start with a value of 3 to 5 and adjust based on your RPM limits
• The semaphore must be shared (global or passed as a parameter), never created locally
• The formula RPM = semaphore * (60 / average_duration) helps size the value