Frontend Beauty
ESC

Type to search the knowledge base.

Output Parsing and Repair

LLM output parsing when structured outputs are unavailable: strip fences, repair JSON, re-ask with validator errors, then give up.

intermediate5 min read
  • parsing
  • json
  • reliability

Trailing commas and ```json fences will outlive every model version you pin. LLM output parsing is the boring layer between a sampled string and a typed object your UI can trust. Prefer native structured outputs. When the provider cannot constrain decoding, or you are stuck on a model that loves markdown, parse then repair, with a cap.

Unbounded “fix the JSON” loops are how a Tuesday typo becomes a token bill. Two repairs. Then fail closed.

Key takeaways

  • Constrained decoding is the default. A repair loop is a fallback, not an architecture.
  • Strip fences, then JSON.parse, then Zod (or JSON Schema). Valid JSON can still be a wrong invoice.
  • Feed validator errors back as a follow-up turn. Do not regex the prose forever.
  • Cap repairs (two is plenty). After that, 422 the client with a stable error code.
  • Never JSON.parse a partial stream. Wait for the object to close, or use a streaming object parser that expects incompleteness.

LLM output parsing as a bounded pipeline

Numbered on purpose. Skip a step and you will invent a third JSON library.

  1. Ask the provider for a schema (response_format, Anthropic structured outputs, AI SDK generateObject). If it holds, stop. Structured output with the AI SDK.
  2. Strip markdown fences, leading “Sure, here is”, trailing commentary.
  3. Parse with JSON.parse. If it throws, optionally run a mechanical repair (trailing commas, smart quotes).
  4. Validate with Zod. Schema-valid is not business-valid: amountCents > paidCents still fails.
  5. Re-ask at most twice, putting the validator message in the next user or tool turn. Then give up.
import { z, type ZodType } from 'zod';

const Invoice = z.object({
  invoiceId: z.string().min(1),
  currency: z.enum(['USD', 'EUR', 'GBP']),
  amountCents: z.number().int().nonnegative(),
});

type Invoice = z.infer<typeof Invoice>;

function stripFence(raw: string): string {
  const trimmed = raw.trim();
  const fenced = trimmed.match(/^```(?:json)?\s*([\s\S]*?)```$/i);
  return (fenced?.[1] ?? trimmed).trim();
}

function tryJson(text: string): unknown {
  try {
    return JSON.parse(text);
  } catch {
    const loosened = text
      .replace(/,\s*([}\]])/g, '$1')
      .replace(/[“”]/g, '"')
      .replace(/[‘’]/g, '"');
    return JSON.parse(loosened);
  }
}

export async function parseWithRepair<T>(
  generate: (hint: string) => Promise<string>,
  schema: ZodType<T>,
  maxRepairs = 2,
): Promise<T> {
  let hint = 'Return a single JSON object. No markdown.';
  let lastErr = 'parse_exhausted';

  for (let attempt = 0; attempt <= maxRepairs; attempt++) {
    const raw = await generate(hint);
    try {
      const value = tryJson(stripFence(raw));
      const parsed = schema.safeParse(value);
      if (parsed.success) return parsed.data;
      lastErr = parsed.error.message;
    } catch (err) {
      lastErr = err instanceof Error ? err.message : 'json_parse';
    }
    hint = `Your previous output failed: ${lastErr}. Return JSON only, no fences.`;
  }

  throw new Error(lastErr);
}

tryJson’s second pass is the “jsonrepair as last resort” move: mechanical, local, no extra model call. If that still throws, you spend a repair turn. A dedicated repair library is fine on step 3. It is not a substitute for step 1.

After the schema succeeds, run your rules:

function assertPayable(inv: Invoice, paidCents: number): Invoice {
  if (inv.amountCents > paidCents) {
    throw new Error('amount_exceeds_paid');
  }
  return inv;
}

The model can emit amountCents: 999999999 in perfect JSON. That is not a parse bug.

What you should repair vs refuse

Symptom Repair? Why
```json wrapper Yes, strip Models will keep doing this
Trailing comma Yes, mechanical Not worth a round trip
Smart quotes, True instead of true Maybe mechanical One pass, then re-ask
Extra keys Ignore or strip via Zod .strict() Decide if unknown keys are an attack
Missing required field Re-ask with the Zod path The model omitted work
Valid JSON, illegal business state Do not repair. 422 Repair would invent money
Partial stream ({"id": "a") Wait, do not parse You will throw every token
HTML / essay with no object Give up after cap Another try will essay harder
Tool call args Same pipeline on args, then auth Parse success is not permission

Streaming is the usual footgun. The UI can paint tokens. Your parser cannot. Buffer the structured payload, or use a parser built for incremental objects. Streaming.

Re-ask as tool feedback, not as prompt poetry

A second generation should see the validator, not “please be careful.” If the provider supports tool results or a user turn, send:

const repairTurn = {
  role: 'user' as const,
  content: `Validation error:\n${lastErr}\nResend the object only.`,
};

Keep temperature low on extraction. A repair pass at 0.8 will invent a prettier, still-wrong field. Pin the same schema on the retry. Do not widen the type “so it passes.”

Retries here are semantic. They are not HTTP 429 retries. Do not stack them with a network retry that also re-fires a side-effecting tool. Retries, timeouts, fallbacks.

Prefer not to write this

Native structured outputs constrain tokens to the schema. You still validate business rules after. The AI SDK’s generateObject is the TypeScript-shaped version of that: Zod in, object out, and you choose what happens on failure instead of hoping.

Write a repair helper when:

  • the only model that can do the task does not expose constrained decoding
  • you are reading an older completion you cannot regenerate
  • a vendor outage forced you onto a fallback model that ignores response_format

Do not write it because a blog post parsed ChatGPT with regex in 2023. That code still fails on a Tuesday, just with more stars in the fence.

Interview answer (out loud)

“I prefer schema-constrained decoding. If I cannot have it, I strip fences, JSON.parse, then Zod, then at most two re-asks that include the validator error. Mechanical repair for trailing commas is allowed. I do not parse partial streams, and I do not let a valid object skip business rules. After the cap I fail closed.”

Further reading (primary sources)

Related guides