How to get structured JSON output from an LLM

Structured output means the model's response is a machine-parseable document conforming to a schema you defined, rather than free prose. In practice that means a JSON object whose keys, types, enumerations, nesting, and required-ness are specified in advance, and whose conformance is verified before your code acts on it.

The four layers, and what each one can and cannot guarantee:

The single most important row is the second-to-last column of layer 2 versus layer 3. Constrained decoding makes the output well-formed. It does not make the output correct. A grammar-constrained model will happily emit {"invoice_total": 0.0, "currency": "GBP"} for an invoice totalling £4,120 — perfectly valid JSON, perfectly conforming to the schema, and completely wrong. Structured output solves a parsing problem, not a truth problem. Candidates and engineers both over-trust it on exactly this point.

L1 — Intuition: describing the shape versus making the wrong shape impossible

There are two fundamentally different ways to get a specific output shape.

You can ask. "Reply with a JSON object containing sentiment and confidence." The model, having seen an enormous quantity of JSON in training, will usually comply. Usually. Sometimes it will wrap the object in a markdown fence. Sometimes it will prepend "Here is the JSON you requested:". Sometimes, on a long or unusual input, it will add a field it thought would be helpful, or trail off mid-object because it hit the token limit.

Or you can prevent. At each generation step the model produces scores over the whole vocabulary (04-04). If the decoder knows the output must be valid JSON, it can set the probability of every token that cannot legally appear next to zero before sampling. After {"sentiment": the only legal continuations are a quote, a digit, t/f/n for literals, {, or [ — so every other token is masked out. The model cannot emit a preamble because a preamble is not a legal JSON document prefix.

That is the whole difference between layer 1 and layer 2: layer 1 changes the distribution; layer 2 changes the support of the distribution.

L2 — Mechanism: how constrained decoding is implemented

The standard implementation compiles your schema or grammar into a state machine, tracks the current state as tokens are emitted, computes the set of tokens legal in that state, and applies a mask to the logits before the sampler runs.

step  partial output           legal next tokens (masked to)     source of constraint
1     (empty)                  {                                 object required
2     {                        "  }                              key or empty object
3     {"                       sentiment  confidence  ...         schema key names
4     {"sentiment              "                                  close the key
5     {"sentiment":            "                                  value is a string
6     {"sentiment": "          positive  negative  mixed          enum values
7     {"sentiment": "positive  "                                  close the value
8     {"sentiment": "positive" ,  }                               next key or end

Note steps 3 and 6. A grammar derived from a full JSON Schema restricts not just punctuation but key names and enum values, which is why a schema-driven constraint is meaningfully stronger than a generic "must be valid JSON" mode. An invented key or an out-of-enum label becomes unsamplable rather than unlikely.

Terminology you should be able to distinguish, because vendors use these labels inconsistently:

Function calling deserves a note because it is the most common way structured output is actually obtained in practice. You declare a function with a typed parameter schema; the model returns a structured call to it. The mechanism underneath is the same schema-constrained generation, and the extra thing you get is routing — which function, with which arguments. If all you want is a typed object out, plain structured output is simpler; if the model must choose among several actions, tool calling is the right shape. 12-13 and the agent material take that further.

L3 — The costs and limits of constraining a decoder

Constrained decoding is not free, and knowing its costs is what separates using it well from using it everywhere.

It cannot improve correctness, and it can interact with quality. Masking tokens forces the model onto a legal path. If the model's preferred continuation was illegal, the constraint pushes it to its best legal alternative — which may be a worse answer, not just a better-formatted one. The practically important instance: forcing an enum removes the model's ability to signal uncertainty. If your enum is positive | negative | mixed and the true answer is "this text is not a review at all," the constraint guarantees you get one of three wrong labels. The remedy is the same escape hatch 05-01 and 05-02 both argued for: put unknown in the enum, and make an optional confidence or reason field available. A constraint with no escape value converts an abstention into a confident error.

It interacts badly with chain-of-thought. A grammar that permits only a JSON object leaves no room for reasoning tokens, which removes the serial computation CoT depends on (05-03). Three ways out, in ascending elegance: two calls (reason, then format); one call with a reasoning string field placed first in the schema so it is generated before the answer fields; or a grammar that permits a reasoning block followed by the object. Field order matters here and is easy to miss — a reasoning field declared after decision is generated after the decision, so it can only rationalise, exactly the failure 05-03 describes.

It costs something at serving time. Compiling a schema to a grammar and computing legal token sets per step is real work, though mature implementations cache compiled grammars and the per-token overhead is usually small relative to the forward pass. On self-hosted stacks this is a configuration and dependency concern, not a free switch.

Deep or exotic schemas strain it. Heavily nested objects, recursive definitions, unions, and unusual regex patterns are where implementations differ most and where you will find provider-specific unsupported-feature errors. Flat, shallow schemas with primitive-typed fields and enums are the reliable core. If your schema needs five levels of nesting, consider splitting into multiple calls.

Truncation still breaks everything. A constrained decoder that hits max_tokens mid-object emits a syntactically incomplete document. The grammar guaranteed only that every prefix was legal, not that generation would reach the end. This is the single most common structured-output failure in production, and the fix is not in the prompt — it is setting max_tokens with headroom and treating a truncation finish reason as a distinct error class.

The comparison table that decides most exam questions in this area.

Two readings the exam rewards.

Nothing in the decoder column ever prevents a wrong value. Every row's "prevents wrong values" cell is No, except the last, and even there only the checkable part — totals that must sum, dates that must be in range, IDs that must exist in your database, citations that must appear verbatim in the source. Semantic correctness is verified against ground truth, not against a schema. This is where 09-07's faithfulness metrics and 13-02's guardrails live.

Validation is not optional even with a perfect constraint. Constrained decoding is a serving-layer feature, and serving layers have bugs, version differences, unsupported schema features, silent fallbacks, and truncation. Your service should be structurally incapable of acting on an unvalidated response. The rule: parse, validate, then act — never parse then act.

All prompts and outputs below are a constructed illustrative example. The failure modes are real and common; the specific outputs are representative rather than logged from a particular model run.

The task. Extract structured fields from a supplier invoice already converted to text (06-01 covers why that conversion is itself a minefield). Downstream, an accounts service inserts the result into a database.

Attempt 1 — ask for JSON in prose.

Extract the invoice details as JSON.

{{invoice_text}}

Representative output — note that the object arrives wrapped in a markdown code fence, with prose on both sides of it:

Here's the extracted invoice data:

~~~json
{
  "supplier": "Northgate Supplies Ltd",
  "invoice_number": "INV-2026-0417",
  "date": "12 July 2026",
  "total": "£4,120.50",
  "vat": "£686.75",
  "line_items": 7
}
~~~

Let me know if you need any other fields extracted!

(The ~~~ markers above stand in for the triple-backtick fence the model actually emitted; they are drawn differently only so this page can display them.)

Six defects in one response, and each maps to a layer:

1. Preamble and postamble. json.loads fails immediately on "Here's the extracted…".
2. Markdown fence. Even after stripping the prose, the fenced-code wrapper must be removed. Every team writes this regex; it is a smell, not a solution.
3. Date in a human format. "12 July 2026" needs parsing and is locale-ambiguous in other forms.
4. Currency amounts as strings with symbols and separators. "£4,120.50" is not a number, and float() on it throws.
5. Key names invented by the model. Your service expects invoice_no and total_gross. It got invoice_number and total. Nothing declared otherwise, so this is your fault, not the model's.
6. No indication of what was not found. If the invoice had no VAT line, would vat be 0, null, "N/A", or absent? Unspecified means all four are possible across calls.

Attempt 2 — state the schema and a literal example (layer 1).

Extract invoice fields from <invoice>. Reply with a single JSON object and
nothing else — no prose, no markdown fences.

Schema:
  supplier_name   string
  invoice_no      string
  issue_date      string, ISO 8601 date (YYYY-MM-DD)
  currency        string, ISO 4217 code (e.g. GBP, EUR, USD)
  total_gross     number, major units, no symbols or separators
  total_vat       number or null if the invoice states no VAT
  line_item_count integer
  extraction_notes string, max 20 words, or "" if nothing notable

If a field is not present in the invoice, use null. Do not infer or calculate a
value that is not stated. If the document is not an invoice, reply with
{"error": "not_an_invoice"}.

Example of a valid response:
{"supplier_name":"Acme Ltd","invoice_no":"A-1","issue_date":"2026-01-31","currency":"GBP","total_gross":120.0,"total_vat":20.0,"line_item_count":3,"extraction_notes":""}

<invoice>
{{invoice_text}}
</invoice>

This is a large improvement and it is worth naming exactly what each addition bought. Explicit key names remove the naming guesswork. Named standards — ISO 8601, ISO 4217 — remove format ambiguity far more compactly than describing the format would. "Major units, no symbols or separators" pre-empts the "£4,120.50" problem. The null rule and the "do not infer" instruction together attack the highest-risk failure in extraction: a model that computes a plausible VAT figure the invoice never stated. And {"error": "not_an_invoice"} is the escape hatch, without which a purchase order gets extracted into confident nonsense.

The one literal example is one-shot prompting from 05-01, aimed at format. It is worth its tokens here because it pins whitespace, quoting, and number style in a way prose cannot.

Representative output: usually correct now. Usually. On a scanned invoice with a garbled total, you may still see a fence, or a helpful "total_net" field you never asked for.

Attempt 3 — constrain the decoder (layer 2). Supply the same shape as a JSON Schema to a serving stack that supports schema-constrained generation:

{
  "type": "object",
  "additionalProperties": false,
  "required": ["supplier_name","invoice_no","issue_date","currency",
               "total_gross","total_vat","line_item_count","extraction_notes"],
  "properties": {
    "supplier_name":    {"type": ["string","null"]},
    "invoice_no":       {"type": ["string","null"]},
    "issue_date":       {"type": ["string","null"], "pattern": "^\\d{4}-\\d{2}-\\d{2}$"},
    "currency":         {"type": ["string","null"], "enum": ["GBP","EUR","USD",null]},
    "total_gross":      {"type": ["number","null"]},
    "total_vat":        {"type": ["number","null"]},
    "line_item_count":  {"type": ["integer","null"]},
    "extraction_notes": {"type": "string", "maxLength": 200}
  }
}

additionalProperties: false is the clause that stops invented fields. required listing every key stops silent omissions — combined with nullable types, "not found" is expressed as an explicit null rather than an absent key, which is a much easier contract for your service to consume. The pattern on issue_date and the enum on currency push two whole classes of malformed value out of the sampler's reach.

Attempt 4 — validate and enforce business rules (layer 3).

from pydantic import BaseModel, Field, field_validator

class Invoice(BaseModel):
    supplier_name: str | None
    invoice_no: str | None
    issue_date: datetime.date | None
    currency: Literal["GBP", "EUR", "USD"] | None
    total_gross: float | None = Field(ge=0)
    total_vat: float | None = Field(ge=0)
    line_item_count: int | None = Field(ge=0)
    extraction_notes: str = ""

    @field_validator("total_vat")
    @classmethod
    def vat_not_above_gross(cls, v, info):
        gross = info.data.get("total_gross")
        if v is not None and gross is not None and v > gross:
            raise ValueError("total_vat exceeds total_gross")
        return v

Then the checks a schema cannot express, which are where the real errors live:

def business_checks(inv: Invoice, source_text: str) -> list[str]:
    problems = []
    if inv.invoice_no and inv.invoice_no not in source_text:
        problems.append("invoice_no not found verbatim in source")   # hallucinated id
    if inv.total_gross is not None and f"{inv.total_gross:,.2f}" not in source_text \
       and f"{inv.total_gross:.2f}" not in source_text:
        problems.append("total_gross not found in source")            # invented total
    if inv.issue_date and inv.issue_date > datetime.date.today():
        problems.append("issue_date is in the future")
    if inv.currency is None and inv.total_gross is not None:
        problems.append("amount without currency")
    return problems

The two verbatim-presence checks are the highest-value assertions in this whole example, and they are free. An extraction task has a property most generation tasks lack: every extracted value should appear in the source document. Checking that turns "did the model hallucinate this number" from an unanswerable question into a string search. Use it wherever the task is extraction, and note it is the same trick as the verbatim-citation assertion in 05-04.

Attempt 5 — handle the failures (layer 4).

def extract(invoice_text: str, attempt: int = 0) -> Invoice | Escalation:
    raw, meta = call_model(TEMPLATE_V4, {"invoice_text": invoice_text})

    if meta.finish_reason == "length":
        # Truncated: not a prompt problem. Raise the ceiling or split the input.
        return Escalation("truncated", meta)

    try:
        inv = Invoice.model_validate_json(raw)
    except ValidationError as e:
        if attempt == 0:
            # One retry, with the specific error handed back. Not a bare retry.
            return extract_with_repair(invoice_text, raw, str(e))
        log.warning("schema_invalid", hash=meta.prompt_hash, error=str(e))
        return Escalation("schema_invalid", meta)

    if problems := business_checks(inv, invoice_text):
        log.warning("business_rule_failed", problems=problems, hash=meta.prompt_hash)
        return Escalation("needs_review", meta, problems=problems)

    return inv

Four properties of that handler are worth stating as rules.

Truncation is its own error class. finish_reason == "length" means the document was cut off, and no amount of retrying the same prompt fixes it. The remedies are a higher max_tokens, a smaller schema, or splitting the input — all different from a validation failure.

Retry once, with the error text included. A bare retry re-rolls the dice; a repair retry ("your previous response failed validation with: total_vat exceeds total_gross. Return a corrected object.") gives the model the information it needs. Cap it at one or two attempts: unbounded retries turn a bad input into a cost incident and a latency spike.

Never repair by regex. Stripping fences, patching trailing commas, and balancing braces is a tempting local fix that hides the real defect and eventually corrupts data silently. If you are writing a JSON repair function, the correct response is to add layer 2 or fix the schema.

Escalate rather than guess. A response that fails validation twice must become a visible, counted event with a human path — not a None that flows downstream. And count these: the rate of schema_invalid, truncated, and needs_review per template version is one of the highest-signal metrics an LLM service has (12-14). A step change in that rate after a deploy tells you the deploy broke something, faster than any quality metric will.

Two schemas can express the same information and differ enormously in how reliably a model fills them. These rules are the difference, and they cost nothing to follow.

When to use which layer:

Objective 4.7 — "Write software components or scripts under the supervision of a senior team member" owns this most directly: an LLM call inside a service is a software component, and making its output parseable is the component's contract. Objective 4.2 / 1.3 (build LLM use cases such as RAG, chatbots, summarizers) reaches it because every one of those use cases eventually needs a machine-readable field — a classification, a set of citations, a routing decision. Objective 4.5 (monitor functioning of software processes) covers counting schema-failure rates. Objective 1.9 supplies the underlying prompt-engineering principle, since output-format constraints are on the must-know list for it. And Objective 4.4 (identify system, hardware, or software components required to meet user needs) is where constrained decoding as a serving-stack capability lands — whether your inference server supports it is a component-selection question.

The official job-role frame is explicit that the associate does "design/code/test/debug/document applications" and "integrating new AI language models into existing systems" [OFFICIAL]. Integration is exactly this problem: existing systems consume typed data, and a language model emits text.

Phrasings that recur:

• "A service needs reliable JSON from an LLM. Which approach is most reliable?" — constrained/structured decoding plus schema validation. Distractors: ask more firmly, lower the temperature, add examples only, post-process with a regex.
• "What does JSON mode guarantee?" — syntactic validity. Not correct values, and in the strict sense not key names either.
• "Constrained decoding prevents…" — invalid syntax and out-of-schema tokens. It does not prevent wrong values.
• "Where should output validation happen?" — in your application code, against the schema, before acting.
• "A model returns valid JSON with an incorrect total. What went wrong?" — a semantic error; structured output does not address correctness.
• "Why include unknown in an enum?" — so abstention is representable; otherwise the constraint forces a wrong label.
• "Which decoding parameter setting reduces format variation?" — a low temperature helps (04-05), but it is not a substitute for a constraint. Watch for options that present it as the complete answer.

Distractor families:

No. JSON mode and schema-constrained decoding guarantee form, never content.

Concretely, a schema-constrained response is guaranteed to parse, to have the keys you declared, to have values of the declared types, and to draw enum fields from the declared set. Within those bounds it can still be wrong in every way that matters: a total that appears nowhere in the source document, a date that is plausible and not the invoice's, a sentiment of positive on a scathing review, an unknown where the answer was clearly present.

This is the highest-value idea in the lesson because it is where over-trust concentrates. A well-typed object looks authoritative in a way prose does not. Engineers who would scrutinise a paragraph of model output will insert a validated object into a database without a second glance, and the schema gave them no license for that confidence.

What actually addresses correctness is a different set of tools, layered on top:

• Verbatim-presence checks for extraction: every extracted value should appear in the source. Cheap, deterministic, and high-yield.
• Cross-field consistency: VAT not above gross, end date after start date, percentages summing to 100.
• Referential checks: the ID exists in your database; the currency is one you trade in.
• Grounding and citation when the content comes from retrieval, so the claim can be traced to a passage (07-09, 09-07).
• Human review for irreversible or high-value actions, triggered by the escalation path in §4.
• A measured eval set with per-field accuracy, so you know your actual error rate rather than assuming it (01-08, 09-01).

The one-sentence version, worth memorising for the exam: constrained decoding is a parsing guarantee, not a truth guarantee.

Yes, materially — and it does not replace the prompt.

Constrained decoding is strictly stronger on syntax because it operates on the sampler rather than on the model's preferences: it masks illegal tokens so an invalid document cannot be produced, where a prompt merely makes an invalid document unlikely. On any endpoint where the shape of the response is load-bearing, that difference is the difference between a service and a demo.

But keep stating the schema and its semantics in the prompt anyway, for four reasons. The constraint tells the model what shape to fill and nothing about what the fields mean — extraction_notes needs an explanation the grammar cannot carry. Rules the grammar cannot express still need saying: "do not infer a value that is not stated," "use null rather than guessing," "quote verbatim." A prompt-stated schema is your fallback when you switch providers, hit an unsupported schema feature, or run on a stack without constraint support. And the prompt is where your escape-hatch policy is explained even when the escape value itself is in the enum.

There is also a quality consideration worth holding in mind. Because a constraint forces the model onto a legal path, a schema that leaves no room for the true answer produces a confidently wrong legal answer. That is not an argument against constraints; it is an argument for schemas with escape values and nullable fields, designed as §5 describes. The prompt and the constraint are doing different jobs: the prompt supplies intent and semantics, the constraint supplies enforcement.

With a defined, counted, bounded path — never with a swallowed exception.

Classify the failure first, because the remedies differ. Truncation (finish_reason == "length") means the ceiling was too low or the input too large; retrying the same call reproduces it. A syntax or schema violation means the constraint was absent, unsupported, or bypassed. A business-rule violation means the document was well-formed and wrong, which is the case most likely to need a human.

Retry once, informatively. Feed back the specific validation error and ask for a corrected object. Cap at one or two attempts. Unbounded retry loops are how a handful of pathological inputs become a latency and cost incident.

Never silently repair or silently drop. Both convert a visible failure into invisible data corruption. If you cannot get a conforming answer, the request has failed, and failing loudly is the correct behaviour.

Escalate with context. The escalation record should carry the prompt template name and version, the rendered-prompt hash, the model revision and decoding parameters, the raw response, and the validation error (05-04). That set is what makes the failure diagnosable weeks later.

Count and alarm. Emit metrics for schema_invalid, truncated, repair_succeeded, escalated, and unknown_enum_rate, broken down by template version. These are leading indicators: they move immediately after a bad deploy or a provider-side model change, well before any quality metric does. Alert on the step change, not the absolute level (12-14).

Fail safe, not open. For any action with real consequences — a payment, a deletion, a message to a customer — the default on validation failure must be to do nothing and escalate. A structured-output failure is exactly the situation in which a system should decline to act.

1. Four layers, used together: prompt-stated schema, constrained decoding, schema validation in code, defined failure handling.
2. A format requested in prose is a request. Only a decoding constraint makes an invalid document unsamplable.
3. Constrained decoding guarantees form, never content. Valid JSON can be entirely wrong.
4. Validate in your own code regardless. Serving layers have bugs, gaps, silent fallbacks, and truncation.
5. Design the schema for the model: flat, self-describing key names, enums for closed sets, primitives for numbers, named standards for formats.
6. Require every key and allow null, and forbid additional properties. "Not found" should be explicit.
7. Every enum needs an escape value. A constraint with no legal abstention manufactures confident errors.
8. Reasoning fields go first, or in a separate call. Left-to-right generation means a field after the answer can only rationalise.
9. Truncation is its own failure class, fixed with token headroom and bounded free-text fields, not with retries.
10. Retry once with the error, then escalate. Never regex-repair, never swallow, never fail open on a consequential action.
11. Add verbatim-presence and cross-field checks — the cheapest hallucination detection available for extraction tasks.
12. Version the schema with the template and count schema-failure rates per version as a leading production indicator.

You have now spent a whole module on what a prompt can specify: how many examples to give, how to lay out instruction and context and format, when reasoning earns its tokens, how to version and test the whole thing, and how to make the output a contract your code can consume. Which raises the question the module has been deferring — the one you now have enough vocabulary to ask properly. Some problems do not yield to any prompt, however well engineered. Missing knowledge is one. Sustained format and style compliance at scale is another. Next: 05-06 gives you a deliberately simple first decision rule for choosing between prompting, RAG, and fine-tuning, and states its own error bars up front: it is roughly right, it underrates fine-tuning for format and style compliance, and you cannot yet measure the exception. 11-08 returns to it with the full rule once you can.