Your AI Returns JSON, Until the One Time It Doesn't

The output that’s valid until it isn’t

There’s a specific kind of bug that only LLM-powered systems have: the data contract that holds 99% of the time and then doesn’t. Traditional code that returns a JSON object returns the same shape every time, it’s deterministic. An LLM asked to return JSON returns something that usually parses, usually has the right keys, usually uses the right types. “Usually” is the entire problem. Your parser, your database insert, your next service in the chain were all written assuming the contract is a contract. It’s a suggestion the model mostly follows.

When it doesn’t follow, the failure is rarely graceful. A malformed object throws a parse exception and 500s the request, or, worse, it parses but has a subtly wrong field, so a bad value flows downstream and corrupts a record nobody notices until much later. The demo never shows this because the demo is one or two inputs. Production is millions, and the long tail is where the malformed outputs live.

Two failure layers: syntax and schema

Structured-output failures come in two distinct layers, and conflating them is why teams under-test. The first layer is syntactic validity, is it parseable JSON at all? The second, deeper layer is schema conformance, does the parseable JSON match the structure you actually require?

This distinction is exactly where “JSON mode” misleads people. JSON mode guarantees the output is syntactically valid, it will parse, but it does not guarantee it matches your schema. You can get perfectly parseable JSON with the wrong types, missing required fields, hallucinated enum values, or extra keys your consumer doesn’t expect. The model satisfied the letter of “return JSON” and violated the contract you cared about.

The numbers make the gap vivid. Prompt-only JSON extraction without constrained decoding fails at roughly 8-15% of calls in production systems processing millions of requests, and on complex schemas adherence drops further; by contrast, OpenAI’s strict Structured Outputs mode achieves syntactic and schema conformance with a failure rate below 0.1% (Structured Output Reliability in Production). Constrained decoding, Structured Outputs, or open tools like Outlines, Guidance, and XGrammar, constrains token sampling to only schema-permitted tokens and reaches near-100% structural conformance. That’s the right default for anything that feeds a typed consumer.

The third layer constrained decoding can’t fix

Here’s the catch that even teams who adopt Structured Outputs miss: constrained decoding guarantees the shape is correct, not that the values are. The grammar forces valid JSON matching your schema, correct types, all required fields, only permitted enums. It cannot force the values to be right. The model can emit a beautifully schema-valid object with a fabricated order ID, a date that contradicts the input, an amount off by a decimal, or a category that’s grammatically a valid enum but semantically the wrong choice, systems routinely emit valid JSON that is wrong around 30% of the time even at near-100% format reliability (Structured Output Isn’t Reliable Output).

Validate all three layers in your eval

# Three-layer structured-output validation
from jsonschema import validate, ValidationError
import json

def validate_output(raw, schema, semantic_checks):
    # Layer 1: syntactic, does it parse at all?
    try:
        obj = json.loads(raw)
    except json.JSONDecodeError as e:
        return {'layer': 'syntax', 'ok': False, 'error': str(e)}

    # Layer 2: schema, right types, required fields, valid enums, no extras
    try:
        validate(instance=obj, schema=schema)   # strict: additionalProperties=False
    except ValidationError as e:
        return {'layer': 'schema', 'ok': False, 'error': e.message}

    # Layer 3: semantic, are the VALUES actually right vs. the input/ground truth?
    for name, check in semantic_checks.items():
        if not check(obj):
            return {'layer': 'semantic', 'ok': False, 'error': f'failed {name}'}

    return {'layer': 'all', 'ok': True, 'obj': obj}

# Run it across the eval set and gate per-layer
def structured_output_gate(model, eval_set, schema):
    layers = Counter()
    for case in eval_set:
        r = validate_output(model(case.input), schema, case.semantic_checks)
        layers[r['layer'] if not r['ok'] else 'pass'] += 1

    n = len(eval_set)
    syntax_ok = 1 - layers['syntax'] / n
    schema_ok = 1 - (layers['syntax'] + layers['schema']) / n
    # With constrained decoding, syntax+schema should be ~100%; if not, fix the config
    assert schema_ok >= 0.999, f"schema conformance only {schema_ok:.2%}"
    # Semantic correctness is the real quality bar
    semantic_ok = layers['pass'] / n
    assert semantic_ok >= 0.95, f"semantic correctness {semantic_ok:.1%}"
    return {'syntax': syntax_ok, 'schema': schema_ok, 'semantic': semantic_ok}

Defense in depth at runtime, too

Eval catches regressions before release; runtime handling catches the residual tail in production. Even at 99.9% conformance, a high-volume system will hit malformed outputs, so the consumer must never assume. Use constrained decoding wherever the provider supports it (turn structural failures from 35% to ~0%); validate every output against the schema at the boundary before it touches downstream systems; and on failure, retry with the validation error fed back to the model, fall back to a safe default, or route to a dead-letter queue, never let an unvalidated object proceed. The combination of constrained generation plus boundary validation plus a graceful failure path is what turns “mostly returns JSON” into a reliable contract.

The bottom line

“Our AI returns JSON” is true right up until the one response that doesn’t, and in a deterministic-downstream pipeline that one response is an outage or a corrupted record. The failure has three layers: syntactic validity (JSON mode covers this), schema conformance (only constrained decoding reliably covers this, lifting complex-schema adherence from ~35-40% to ~100%), and semantic correctness (nothing structural covers this, the values can be schema-valid and wrong). Validate all three in your eval, gate releases on schema and semantic conformance, and add runtime validation with a graceful failure path. The demo works because the demo is small. Production is large, and large is where the malformed object is waiting.