Self-Correcting LLM Extraction Loops

Building resilient validation pipelines that catch Pydantic ValidationError exceptions, format the exact schema error message into a repair prompt, and re-query the model to achieve 99.9% extraction reliability.

Interactive Systems Visualizer

Pydantic v2 Core Rust Validation & Error Recovery

Raw LLM Output String:

{
  "query": "vector index latency",
  "confidence": 0.94,
  "is_urgent": true
}

Pydantic Core Parser Result:

STATUS: VALID

Parsed successfully by pydantic-core in 0.08ms

Key Systems Engineering Principles

Models rarely make the same structural mistake when shown their exact JSON path error.
Repair prompt injection: append the invalid JSON and the Pydantic ValidationError trace to the conversation.
Limits retry count (typically max 2-3 attempts) to bound cost and latency.

Mental Model Analogy

“A self-correcting loop is a compiler telling a programmer: "Syntax error on line 4: missing comma" instead of silently crashing the whole operating system.”

Production Pitfall to Avoid

Retrying the exact same prompt repeatedly without telling the LLM what validation error it produced, resulting in identical failures.

production_pattern.py

from pydantic import BaseModel, ValidationError

class UserProfile(BaseModel):
    age: int
    email: str

def parse_with_repair(raw_output: str, model_client, retries=2):
    for attempt in range(retries):
        try:
            return UserProfile.model_validate_json(raw_output)
        except ValidationError as e:
            repair_prompt = f"Fix this JSON: {raw_output}. Errors: {e.errors()}"
            raw_output = model_client(repair_prompt)
    raise RuntimeError("Failed to extract valid schema after retries")

Previous Subtopic#19 Structured LLM Outputs & JSON Schema Extraction Next Subtopic #21 Generators & Lazy Evaluation for Massive Datasets

Loading Python for AI Engineering...