JSON Mode Limitations and Error Handling — Structured Output: JSON and Schemas

Valid Does Not Mean Correct

JSON Mode guarantees syntactically valid JSON, not semantically correct JSON. This distinction might seem subtle during development, but it becomes critical in production. JSON that parses without errors but contains unexpected keys creates exactly the kind of bug that takes days to track down, because no exception triggers when the problem actually occurs.

No Schema Guarantees

The mechanism knows nothing about your schema: it has never seen the structure your code expects. The model might omit a field you need, add one you didn’t request, rename a key—firstName instead of first_name—or change a type by returning "42" as a string when you expected the number 42. None of this is invalid JSON.

# You expect this:
# {"name": "Alice", "age": 30, "city": "Paris"}

# But you might receive:
# {"nom": "Alice", "âge": "trente ans", "ville": "Paris", "pays": "France"}

The JSON is valid, but the structure doesn’t match what your code expects. This is the fundamental difference from Custom Structured Outputs.

Mandatory Client-Side Validation

Since the schema isn’t guaranteed, validation falls on you. The minimal pattern does three things in order: it attempts parsing, verifies the root is an object (not a list or scalar), then checks for required keys. Each failure raises a ValueError with a message you can use in your logs.

import json
from typing import Optional

def parse_and_validate(raw_content: str, required_keys: list[str]) -> dict:
    """Parse JSON and verify required keys."""
    try:
        data = json.loads(raw_content)
    except json.JSONDecodeError as e:
        raise ValueError(f"Invalid JSON: {e}")

    if not isinstance(data, dict):
        raise ValueError(f"Expected JSON object, received {type(data).__name__}")

    missing = [k for k in required_keys if k not in data]
    if missing:
        raise ValueError(f"Missing keys: {missing}")

    return data

# Usage
try:
    result = parse_and_validate(
        response.choices[0].message.content,
        required_keys=["name", "category", "priority"]
    )
    process_ticket(result)
except ValueError as e:
    log_error(f"Invalid output: {e}")
    fallback_processing(raw_content)

The TypeScript version goes a step further by verifying, beyond key presence, that the returned category belongs to the allowed list. This is the check most often missing, and the one that prevents a ticket classified as "urgent_billing" from silently slipping through your routing.

interface TicketClassification {
  category: string;
  priority: string;
  sentiment: string;
  summary: string;
}

function validateTicket(raw: string): TicketClassification {
  const data = JSON.parse(raw);

  const requiredKeys = ["category", "priority", "sentiment", "summary"];
  const missing = requiredKeys.filter((k) => !(k in data));

  if (missing.length > 0) {
    throw new Error(`Missing keys: ${missing.join(", ")}`);
  }

  const validCategories = ["billing", "account", "technical", "feature_request", "other"];
  if (!validCategories.includes(data.category)) {
    throw new Error(`Invalid category: ${data.category}`);
  }

  return data as TicketClassification;
}

Three Failures You Will Encounter

The first is truncated JSON. If the response hits the max_tokens limit, it gets cut off mid-stream and becomes invalid—the closing brace never arrives. You can spot the symptom before even parsing: finish_reason equals "length" instead of "stop".

response = client.chat.complete(
    model="mistral-small-latest",
    messages=messages,
    response_format={"type": "json_object"},
    max_tokens=50  # Too small for complete JSON
)

# response.choices[0].finish_reason will be "length" instead of "stop"
if response.choices[0].finish_reason == "length":
    print("Warning: Truncated response, increase max_tokens")

Always check finish_reason: if it equals "length", the response is incomplete, regardless of what the rest of the content says.

The second failure concerns types. An age might arrive as a string, a price as an integer when you expect a float. Rather than trusting the output, wrap access in a helper that attempts conversion and falls back to a default value if it fails.

def safe_get_int(data: dict, key: str, default: int = 0) -> int:
    """Extract an integer, with conversion if needed."""
    value = data.get(key)
    if value is None:
        return default
    if isinstance(value, int):
        return value
    if isinstance(value, float):
        return int(value)
    if isinstance(value, str):
        try:
            return int(value)
        except ValueError:
            return default
    return default

# Usage
age = safe_get_int(data, "age", default=0)

The third is key renaming. With a French prompt, the model might readily switch phone to telephone, or name to nom. A synonym mapping absorbs these variations without requiring you to retry the call.

def normalize_keys(data: dict, key_mapping: dict) -> dict:
    """Normalize keys using a synonym mapping."""
    normalized = {}
    for standard_key, aliases in key_mapping.items():
        for alias in aliases:
            if alias in data:
                normalized[standard_key] = data[alias]
                break
    return normalized

# Synonym mapping
mapping = {
    "name": ["name", "nom", "full_name", "fullName"],
    "email": ["email", "mail", "e-mail", "emailAddress"],
    "phone": ["phone", "telephone", "tel", "phoneNumber"],
}

normalized = normalize_keys(raw_data, mapping)

Retry with Feedback

When validation fails despite your best efforts, the best response isn’t to retry the exact same call: it’s to tell the model what went wrong. By feeding its own response back into the conversation along with the error message, you give it the context needed to self-correct, usually by the second attempt.

import json

def call_with_retry(client, messages, max_retries=3):
    """Call with retry and error feedback."""
    for attempt in range(max_retries):
        response = client.chat.complete(
            model="mistral-large-latest",
            messages=messages,
            response_format={"type": "json_object"},
            temperature=0
        )

        raw = response.choices[0].message.content
        try:
            data = json.loads(raw)
            validate_schema(data)  # Your validation function
            return data
        except (json.JSONDecodeError, ValueError) as e:
            if attempt < max_retries - 1:
                # Add error feedback to context
                messages.append({"role": "assistant", "content": raw})
                messages.append({
                    "role": "user",
                    "content": f"Your response is invalid: {e}. Correct it and return valid JSON."
                })
            else:
                raise RuntimeError(f"Failed after {max_retries} attempts: {e}")

The Signal to Change Approach

If you find yourself writing extensive validation code, that’s a strong signal it’s time to migrate to Custom Structured Outputs. Three typical thresholds confirm this: beyond five fields to validate, the control function becomes longer than the call itself; once types get complex with nested lists or enums, manual validation becomes inseparable from the bugs it introduces; and for any critical production pipeline, delegating constraints to the schema is simply safer than reconstructing them by hand.

Key Takeaways

JSON Mode guarantees valid JSON, not a valid schema
Always validate output client-side: check for required keys, correct types, and values within bounds
Check finish_reason to detect truncated responses
Implement a retry mechanism with error feedback
If validation becomes complex, migrate to Custom Structured Outputs