Validation Strategy Decision Framework
| Scenario | Recommended Approach | Why |
|---|---|---|
| OpenAI with Pydantic schemas | response_format with strict: true | Guaranteed valid JSON from constrained decoding — no validation needed |
| OpenAI with business logic validation | Instructor + Pydantic validators | Auto-retry feeds validation errors back to model |
| Open-source models (Ollama, vLLM) | Instructor or custom JSON extraction + Pydantic | Open-source models do not support constrained decoding via API |
| Multi-provider setup | Instructor (provider-agnostic) | Works with OpenAI, Anthropic, Ollama, etc. via same interface |
| Legacy system / no library | Fallback parser chain | Try direct JSON, code block, embedded, key-value, then LLM repair |
| Safety-critical outputs | LLM-as-validator + deterministic checks | Belt-and-suspenders: programmatic validation catches structure, LLM catches semantics |
| High-volume batch processing | strict: true + schema (no retries) | Retries are too expensive at scale; fail fast and log for review |
Instructor for Validated Outputs
Instructor is the most popular library for getting structured outputs from LLMs, and for good reason: it wraps the OpenAI client with Pydantic validation and automatic retries. When the model returns invalid data, Instructor automatically sends the validation error back to the model and asks it to fix the output. Think of it as a patient teacher who keeps handing back the assignment until the student gets it right.Basic Usage
Complex Nested Structures
Retry Logic with Validation
This is where Instructor really shines. You define Pydantic validators that encode your business rules, and when the model violates them, Instructor sends the validation error back as context so the model can self-correct. The combination ofinstructor.max_retries (validation retries) and tenacity.retry (network retries) gives you a robust two-layer retry strategy.
Custom Validation Strategies
Sometimes you do not need the model to produce structured output — you need to extract structured data from whatever the model produces. This is common when you are working with models that do not support structured outputs natively (open-source models via Ollama, for example) or when you are post-processing outputs from a pipeline you do not fully control.Regex-Based Extraction
JSON Extraction and Validation
This is the Swiss Army knife of LLM output parsing. Models love to wrap JSON in markdown code blocks, prefix it with “Here’s the data:”, or include trailing commentary. The extractor below handles all of these cases gracefully by trying multiple parsing strategies in order.LLM-Based Validation
This is the “quis custodiet ipsos custodes” pattern — using an LLM to watch another LLM. It sounds circular, but it works because the validator model operates on a simpler, more constrained task (binary judgment) than the generator model (open-ended creation). The key insight: validation is easier than generation. A model that cannot reliably write a factually accurate summary can still reliably judge whether a given summary is factually accurate, especially when given the source material. Use an LLM to validate another LLM’s output.Fallback Parsing Strategies
Real-world LLM outputs are messy. Sometimes the model returns clean JSON. Sometimes it wraps it in a code block. Sometimes it returns key-value pairs. And sometimes it returns something entirely unexpected. The fallback parser below tries each parsing strategy in order from cheapest to most expensive, and stops at the first success. The final fallback — calling another LLM to fix the output — is expensive but remarkably effective as a last resort.Validation Best Practices
- Always validate LLM outputs before using them
- Use Pydantic for type-safe structured extraction
- Implement fallback strategies for robustness
- Consider LLM-based validation for complex checks
- Log validation failures for debugging and improvement
Validation Edge Cases
Edge case — partial JSON from streaming: When streaming responses, the model sends tokens incrementally. If you try to validate mid-stream, you will always get invalid JSON. Buffer the full response before validating, or use a streaming JSON parser likeijson that can process incomplete JSON incrementally.
Edge case — Unicode and encoding issues: Models sometimes produce Unicode escape sequences (\u00e9 instead of the actual character) or smart quotes that break strict JSON parsers. Always normalize Unicode before parsing: text.encode('utf-8').decode('utf-8').
Edge case — the model refuses to produce output: Content policy violations result in a refusal message instead of your expected JSON. Check finish_reason — if it is "content_filter", the response is not your structured output. Handle this gracefully rather than passing a refusal message to your JSON parser.
Edge case — null vs. missing fields: There is a meaningful difference between {"email": null} and {} (no email key). Pydantic’s Optional[str] = None treats both the same, but your business logic might care. Use Pydantic’s model_config = ConfigDict(extra="forbid") to catch unexpected fields, and explicitly distinguish between None (user has no email) and missing (model forgot to extract it).
Edge case — numeric precision: LLMs sometimes return "price": 19.99 as "price": 19.990000000000002. If you are doing financial calculations on extracted values, round explicitly in your Pydantic validator. Do not trust the model’s floating-point precision.
Practice Exercise
Build a validation system that:- Extracts structured data from free-form LLM responses
- Validates against Pydantic schemas with custom validators
- Implements multiple fallback parsing strategies
- Uses LLM-based validation for complex checks
- Provides detailed error messages for failures
- Handling edge cases and malformed outputs
- Performance optimization for high-volume validation
- Comprehensive logging of validation failures
- Automatic retry and correction strategies