Structured Output Prompts: Get Valid JSON from LLMs with Schemas, Examples, and Recovery Patterns

Overview

Applications often need an LLM to turn unstructured text into data: a support ticket into a category and priority, a meeting transcript into action items, or an invoice into fields that another system can process. Natural-language answers are useful for people, but they are difficult for software to consume safely. Structured output prompts address this gap by specifying the shape and rules of the expected response.

The objective is not simply to force ChatGPT or another model to place braces around its answer. A dependable design treats the output as a data contract with four layers:

1. Schema: the permitted fields, types, nesting, and required values.
2. Prompt instructions: the task definition, constraints, and handling of uncertainty.
3. Generation controls: JSON mode or schema-constrained generation when the selected model and API support them.
4. Application validation: code that checks and safely handles every response before it enters a database or workflow.

These layers solve different problems. A schema can reject an incorrect type, but it cannot always determine whether the model extracted the right person or date. Prompt instructions can explain an ambiguous field, but they do not replace runtime validation. For more advanced integrations, compare this approach with LLM function calling and tool use, where schemas can also define actions and approval steps.

Which output method should you use?

Use the strongest supported generation method available, but keep validation in your application. Provider syntax, supported schema keywords, SDK behavior, and token accounting can change, so verify those details against the documentation for the exact model and endpoint you use.

How to estimate reliability and cost

A useful estimate separates the quality of the initial response from the cost of handling failures. Let:

• R: the number of input records.
• T: average input tokens per record.
• O: average output tokens per record.
• P: the proportion of records that require a retry or repair.
• C_in and C_out: the provider's input and output rates for the selected model.
• V: the proportion of responses that pass schema and business-rule validation on the first attempt.

The approximate first-pass token volume is:

input tokens = R × T
output tokens = R × O

A simple first-pass cost estimate is:

cost = (R × T × Cin) + (R × O × Cout)

If a fraction of records needs another model call, include the expected recovery volume:

adjusted cost ≈ first-pass cost × (1 + P)

This is an estimate, not an invoice. Actual usage can vary because prompts, examples, long source documents, repair messages, and model-specific tokenization all affect the total. Keep input and output rates as configuration values rather than embedding them in code. This makes it easier to recalculate when pricing inputs or model selections change.

For quality, begin with a labeled test set rather than a single successful example. Calculate:

schema pass rate = responses passing structural validation ÷ total responses

Then track business-level accuracy separately. A response can be valid JSON and still contain the wrong category, an invented value, or an incorrect date. For a repeatable evaluation process, use the prompt testing and evaluation checklist and record both pass rates and failure types.

Inputs and assumptions

Reliable JSON begins before the prompt is written. Define what the application actually needs, and avoid asking the model to return fields that no downstream process uses.

1. Design the response schema

For each field, decide its name, type, required status, allowed values, and behavior when the source does not provide an answer. Explicit nullability is often safer than making the model guess.

{
  "type": "object",
  "properties": {
    "category": {"type": "string", "enum": ["billing", "technical", "account", "other"]},
    "priority": {"type": "string", "enum": ["low", "normal", "high", "urgent"]},
    "summary": {"type": "string"},
    "customer_id": {"type": ["string", "null"]},
    "confidence": {"type": "number", "minimum": 0, "maximum": 1}
  },
  "required": ["category", "priority", "summary", "customer_id", "confidence"],
  "additionalProperties": false
}

Use stable machine-readable names instead of labels that may change. Choose numbers for quantities that will be calculated, booleans for genuine yes-or-no states, and enums for small controlled vocabularies. If a field can contain several values, define whether an empty array or null is the correct representation.

2. Define uncertainty and source boundaries

Tell the model what to do when evidence is missing. A useful rule is: “Use only the supplied text. If a value is not stated or cannot be determined, return null. Do not infer identity, dates, amounts, or causes.” This is more actionable than a general instruction to “be accurate.”

If the model receives retrieved documents, include source identifiers in the schema where traceability matters. A RAG workflow should distinguish between an extracted fact and an unsupported response; the RAG tutorial provides related guidance on retrieval and verification.

3. Add focused few-shot examples

Examples are most useful when they demonstrate edge cases: missing fields, conflicting dates, multiple possible categories, and text that is irrelevant to the task. Keep each example aligned with the current schema. If the schema changes, update the examples in the same change set.

System:
Extract support-ticket fields from the supplied text.
Return only an object matching the response schema.
Use null when customer_id is not stated. Do not invent values.

Example input:
"The dashboard shows a timeout when I export a report."
Example output:
{"category":"technical","priority":"normal","summary":"Customer reports a timeout during report export.","customer_id":null,"confidence":0.91}

User:
{{ticket_text}}

4. Validate in two stages

First, parse the response as JSON. Second, validate the parsed object against the schema and business rules. Business rules might require that a high-priority ticket includes an explanation, that a date falls within an acceptable range, or that an identifier matches a known record.

raw = call_model(prompt)

try:
    data = parse_json(raw)
except ParseError:
    record_failure("invalid_json", raw)
    send_to_recovery_or_review(raw)
else:
    errors = validate_schema(data)
    errors += validate_business_rules(data)
    if errors:
        record_failure("validation_error", errors)
        send_to_recovery_or_review(data, errors)
    else:
        persist(data)

Never treat parsing success as proof that the extraction is correct. Store the original input, model identifier, prompt version, schema version, validation result, and recovery path so that failures can be reproduced.

Worked examples

Example: extracting action items

Suppose a meeting-notes workflow needs an array of action items. A practical contract might require task, owner, due_date, and evidence. Set owner and due_date to nullable fields because meetings frequently omit them. Require evidence to contain the relevant phrase from the notes, which gives a reviewer a quick way to check the extraction.

For an input containing “Jordan will send the revised draft by Friday,” the expected result could be:

{
  "action_items": [
    {
      "task": "Send the revised draft",
      "owner": "Jordan",
      "due_date": "Friday",
      "evidence": "Jordan will send the revised draft by Friday."
    }
  ]
}

Do not silently convert “Friday” into a calendar date unless the meeting date and timezone are available. A deterministic application step can resolve relative dates later, with explicit assumptions.

Recovery pattern for malformed or incomplete output

Recovery should be narrow and observable. First, retry the original request only when the failure may be transient. If the output is almost valid but contains an extra explanation, a constrained repair step can ask the model to transform the supplied text into the existing schema without changing its factual content. Validate the repaired result again.

Repair instruction:
Convert the following candidate into the supplied JSON schema.
Do not add facts, change values, or infer missing information.
Use null for unsupported fields.
Return only JSON.

Validation errors:
{{errors}}
Candidate:
{{candidate}}

Set a maximum number of recovery attempts. Route persistent failures to a review queue, especially when the output triggers payments, account changes, external messages, or other consequential actions. For retry, timeout, and fallback design, see the guide to LLM API error handling.

Estimate a batch before production

Assume a pilot processes 2,000 records, with an average of 900 input tokens and 180 output tokens per record. If 8% of records require one recovery call of similar size, the estimated request volume is equivalent to:

2,000 × 1.08 = 2,160 model calls

The approximate token totals are:

• Input: 2,000 × 900 × 1.08 = 1,944,000 tokens.
• Output: 2,000 × 180 × 1.08 = 388,800 tokens.

Multiply those totals by the current input and output rates for the chosen model. If your repair prompt includes the original candidate and validation errors, its token count may differ; measure it from logs rather than assuming it is identical.

When to recalculate

Revisit the prompt, schema, evaluation set, and cost estimate whenever the inputs change. In practice, that means recalculating after:

• Adding, removing, renaming, or changing the type of a field.
• Changing the source documents, average input length, or expected output detail.
• Switching models, endpoints, SDKs, or structured-output mechanisms.
• Changing the number of few-shot examples or the recovery strategy.
• Seeing a rise in invalid JSON, schema failures, low-confidence results, or human corrections.
• Changing provider pricing inputs, request limits, or model availability.

Keep a small regression set that represents normal cases and difficult edge cases. Run it whenever the prompt or schema changes, then compare structural pass rate, field-level accuracy, recovery rate, latency, and estimated cost. Avoid optimizing only for valid JSON: a perfectly formed object with incorrect values is still a failed extraction.

Finally, version the prompt and schema together, log failures by category, and make human review available for ambiguous records. These habits turn structured output prompts from a formatting trick into a maintainable part of an AI development workflow. For broader workflow design, see AI workflow automation with human-in-the-loop controls and the guide to context window management.