The moment an LLM's output feeds code instead of a human, "usually valid JSON" becomes a bug class: a stray markdown fence, a trailing comma, or an invented field name breaks the parser on the 1-in-50 call, which for an agent loop means a crash mid-task. Schema-constrained generation closes that class — OpenAI's structured outputs enforce conformance to a supplied JSON Schema during decoding, and Anthropic's structured outputs offer the same contract (with tool-input schemas as the long-standing equivalent pattern). The retry-on-parse-failure loop you wrote in 2024 is now the provider's job.
Write schemas that carry intent
The schema is not just a validator — the model reads it as instructions:
- Closed enums over free strings.
"status": "open" | "resolved" | "escalated"eliminates the synonym roulette ("closed", "done", "finished") that breaks downstream switch statements. - Required fields + no additional properties, where the provider's schema subset allows, so drift is impossible in both directions.
- Field descriptions do prompting work. Units, timezone, format, and "null when unknown — never guess" belong on the field, next to where the model decides.
- Model uncertainty explicitly. A nullable field with a "why missing" companion beats forcing a value; forced fields get filled with plausible fabrications.
Shape is not truth
Provider-side conformance guarantees the JSON parses and matches the schema — nothing more. The values can still be wrong, stale, unauthorized, or referentially broken (customer_id that doesn't exist). Treat schema-valid output as untrusted input that parses: semantic validation, range checks, authorization, and existence checks still run in your code before anything acts on it. This matters doubly for agents, where a structured tool call is often one step from a side effect.
Handle the non-happy paths as distinct outcomes
Refusals, truncation (max-token cutoffs mid-object), and provider errors are different failures with different fixes — a refusal should surface, truncation should raise the budget or shrink the schema, an API error should retry. Collapsing them into one catch block turns a diagnosable failure into a mystery. And version your schemas: producers and consumers evolve independently, and a silently-added enum value is a breaking change to the consumer that switches on it.
Sources: OpenAI — Structured outputs · Anthropic — Structured outputs.
Related: tool schema design, tool use, guardrails & safety, determinism & reproducibility.