HTTPX Connection Pooling & HTTP/2 Multiplexing

Utilizing httpx.AsyncClient with persistent TLS connection reuse and HTTP/2 multiplexing to eliminate repeated TCP 3-way handshakes and TLS negotiations on high-throughput LLM API calls.

Interactive Systems Visualizer

CPython Global Interpreter Lock (GIL) Concurrency Engine

When running pure Python bytecode, only ONE thread holds the GIL at any instant. Other threads stall waiting for the lock:

Core #1

ACTIVE (Holding GIL)

Core #2

BLOCKED (Waiting for GIL)

Core #3

BLOCKED (Waiting for GIL)

Core #4

BLOCKED (Waiting for GIL)

Effective Multi-Core Scaling:1.0x (No multi-core speedup)

Key Systems Engineering Principles

Reusing a single AsyncClient instance across requests saves 100ms-300ms of TLS handshake latency per call.
HTTP/2 multiplexing sends concurrent requests over a single TCP socket without head-of-line blocking.
Always configure explicit connect, read, and write timeouts to prevent hung requests during upstream provider outages.

Mental Model Analogy

“Connection pooling is a permanent dedicated phone line between two offices, instead of dialing the operator, negotiating numbers, and verifying identity on every single word.”

Production Pitfall to Avoid

Instantiating a new httpx.AsyncClient() inside every function call, closing sockets immediately and suffering connection exhaustion (TIME_WAIT).

production_pattern.py

import httpx

limits = httpx.Limits(max_keepalive_connections=20, max_connections=100)
timeout = httpx.Timeout(connect=5.0, read=30.0, write=5.0, pool=5.0)

# Singleton client pattern
async def get_client() -> httpx.AsyncClient:
    return httpx.AsyncClient(limits=limits, timeout=timeout, http2=True)

# Usage across entire server lifecycle:
# client = await get_client()
# response = await client.post("https://api.openai.com/v1/...", ...)

Previous Subtopic#14 Token Streaming with Async Generators (async for)Next Subtopic #16 Modern Type Hinting (Generics, Unions, TypeVar)

Loading Python for AI Engineering...

Key Systems Engineering Principles

Reusing a single AsyncClient instance across requests saves 100ms-300ms of TLS handshake latency per call.

HTTP/2 multiplexing sends concurrent requests over a single TCP socket without head-of-line blocking.

Always configure explicit connect, read, and write timeouts to prevent hung requests during upstream provider outages.

import httpx limits = httpx.Limits(max_keepalive_connections=20, max_connections=100) timeout = httpx.Timeout(connect=5.0, read=30.0, write=5.0, pool=5.0) # Singleton client pattern async def get_client() -> httpx.AsyncClient: return httpx.AsyncClient(limits=limits, timeout=timeout, http2=True) # Usage across entire server lifecycle: # client = await get_client() # response = await client.post("https://api.openai.com/v1/...", ...)