Token Streaming with Async Generators (async for)

Consuming and emitting real-time token streams via async generators (yield in async def). Powers modern Server-Sent Events (SSE) and WebSocket endpoints to achieve Time-To-First-Token (TTFT) under 300ms.

Interactive Systems Visualizer

Token Streaming via Async Generator (async for)

// Click "Emit Stream" to test async generator yielding...

Time to First Token (TTFT): ~120msTokens Emitted: 0 / 10

Key Systems Engineering Principles

Async generators yield individual token chunks as they arrive over HTTP chunked transfer encoding.
Reduces perceived user latency: users read generated text immediately rather than waiting 15 seconds for complete synthesis.
Supports cooperative cancellation: if the client disconnects, breaking the async generator loop cleanly terminates upstream GPU generation.

Mental Model Analogy

“Streaming is drinking water from a tap as it flows; non-streaming is waiting for a 5-gallon bucket to fill before taking a single sip.”

Production Pitfall to Avoid

Buffering all stream chunks into a string inside the generator before yielding, completely defeating the purpose of streaming.

production_pattern.py

import asyncio
from typing import AsyncGenerator

async def mock_token_stream(prompt: str) -> AsyncGenerator[str, None]:
    tokens = ["The", " future", " of", " AI", " is", " streaming."]
    for token in tokens:
        await asyncio.sleep(0.08) # Simulate token generation latency
        yield token

async def client_consumer():
    print("Beginning stream: ", end="", flush=True)
    async for chunk in mock_token_stream("hello"):
        print(chunk, end="", flush=True)
    print("
[Stream complete]")

Previous Subtopic#13 Rate-Limiting, Throttling & Async Semaphores Next Subtopic #15 HTTPX Connection Pooling & HTTP/2 Multiplexing

Loading Python for AI Engineering...