Rate-Limiting, Throttling & Async Semaphores

Controlling concurrency using asyncio.Semaphore to enforce vendor TPM (Tokens Per Minute) and RPM (Requests Per Minute) boundaries without getting rejected with HTTP 429 Too Many Requests.

Interactive Systems Visualizer

CPython Global Interpreter Lock (GIL) Concurrency Engine

When running pure Python bytecode, only ONE thread holds the GIL at any instant. Other threads stall waiting for the lock:

Core #1

ACTIVE (Holding GIL)

Core #2

BLOCKED (Waiting for GIL)

Core #3

BLOCKED (Waiting for GIL)

Core #4

BLOCKED (Waiting for GIL)

Effective Multi-Core Scaling:1.0x (No multi-core speedup)

Key Systems Engineering Principles

An asyncio.Semaphore(N) acts as a concurrency gatekeeper, allowing at most N simultaneous calls.
Pairing semaphores with exponential backoff and jitter guarantees smooth recovery during API rate-limit spikes.
Leaky bucket algorithms pace bursts of requests into steady, deterministic request streams.

Mental Model Analogy

“A semaphore is a nightclub bouncer: only 5 patrons are allowed on the dance floor at once. When one leaves, the next in line enters.”

Production Pitfall to Avoid

Launching 500 parallel API calls without a semaphore, instantly exhausting OpenAI/Anthropic rate limits and triggering 429 backoff penalties.

production_pattern.py

import asyncio

sem = asyncio.Semaphore(3) # Max 3 concurrent requests

async def safe_llm_call(task_id: int):
    async with sem: # Waits if 3 tasks are already inside
        print(f"Task {task_id} acquired permit")
        await asyncio.sleep(0.5) # Network I/O
        print(f"Task {task_id} released permit")
        return f"Result {task_id}"

async def main():
    await asyncio.gather(*(safe_llm_call(i) for i in range(9)))

Previous Subtopic#12 Concurrent LLM Fan-Out & Task Gathering Next Subtopic #14 Token Streaming with Async Generators (async for)

Loading Python for AI Engineering...