Frontend Beauty
ESC

Type to search the knowledge base.

Debugging LLM Hallucinations

A hallucination is a fluent claim without support. Debug it as retrieval miss, prompt pressure, sampler noise, or missing refusal path.

intermediate6 min read
  • hallucination
  • debugging
  • quality

The answer cited https://docs.example.com/v3/refunds#windows. The chip looked like every other citation in your UI. The path 404’d. That is an LLM hallucination: a fluent claim with no support in the evidence you provided, the tools you ran, or the world.

Do not start by rewriting adjectives in the system prompt. Split the failure: did we retrieve the wrong thing, did we bully the model into answering, did the sampler wander, or did we forget to allow “I don’t know”? Then verify ids and URLs in code. The UI should not render a footnote it did not check.

Key takeaways

  • A hallucination is an unsupported claim, not a rude tone. Fluency is the disguise.
  • Debug in four buckets: retrieval miss, prompt pressure, sampler noise, missing refusal.
  • Force quote-then-answer, then verify chunk ids and URLs outside the model.
  • Log the packed context, not only the final string, or you cannot replay the bug.
  • Teaching refusal is a product feature. Empty retrieval should not produce a paragraph.

Four sources of an LLM hallucination

Bucket What happened First fix
Retrieval miss Gold evidence never entered the prompt Index, query, filters, k, rerank
Prompt pressure “Always be helpful” beat “only from the docs” Permission to refuse, shorter task
Sampler noise Right prompt, spicy temperature, extra field Lower temperature, structured output
Missing refusal No branch for “we don’t know” Empty-retrieval template, eval on it

These stack. A miss plus a helpful prompt is the usual production incident: the model writes a plausible policy because you asked it not to disappoint.

Retrieval miss. The nearest chunk was a blog post from 2021, or the table split in half, or tenant filters dropped the only good hit. Faithfulness metrics will look fine if you score against what was packed and terrible if you score against what is true. Split those evals (RAG evaluation metrics).

Prompt pressure. Few-shot examples that always answer, a system prompt that says “never say you don’t know,” or a user who writes “just guess the order id.” The decoder is doing what you rewarded.

Sampler noise. Extraction at temperature 0.8 invents keys. Temperature 0 is still not a cross-provider constant, but it is the right direction for JSON. See temperature, top-p, and sampling.

Missing refusal. If the only successful demo you ever showed the model is a full answer, it will emit a full answer. Put refuse cases in the golden set.

Force evidence, then verify it

Ask for claims bound to ids you issued. Then check that those ids exist in this request’s retrieved set. A footnote to chunk #3 is not proof that the sentence is entailed by chunk #3. It is a pointer you can test.

type Claim = { text: string; chunkIds: string[] };

type Packed = { id: string; text: string };

export function auditClaims(claims: Claim[], packed: Packed[]) {
  const byId = new Map(packed.map((c) => [c.id, c]));
  const problems: string[] = [];

  for (const claim of claims) {
    if (claim.chunkIds.length === 0) {
      problems.push(`unsupported: ${claim.text}`);
      continue;
    }
    for (const id of claim.chunkIds) {
      if (!byId.has(id)) problems.push(`unknown-id: ${id}`);
    }
  }
  return problems;
}

export async function deadUrls(markdown: string): Promise<string[]> {
  const found = [...markdown.matchAll(/https?:\/\/[^\s)"']+/g)].map((m) => m[0]);
  const dead: string[] = [];

  for (const url of found) {
    try {
      const res = await fetch(url, { method: 'GET', redirect: 'follow' });
      // HEAD is flaky on CDNs. GET and ignore the body if you only care about status.
      if (res.status >= 400) dead.push(url);
    } catch {
      dead.push(url);
    }
  }
  return dead;
}

export function refuseIfEmpty(packed: Packed[], generate: () => Promise<string>) {
  if (packed.length === 0) {
    return {
      text: 'I do not have a document that answers this. Try a more specific question.',
      claims: [] as Claim[],
    };
  }
  return generate();
}

Grounding is entailment, not layout. Citations are the UI. If you cannot check entailment cheaply, check the cheap parts first: unknown ids, dead URLs, SKUs that fail a DB lookup, amounts that fail a sum.

Do not let the model invent URLs at all if you can paste the canonical path from metadata instead. Your retriever already knew the path.

A replay you can actually run

“The bot was weird” is not a ticket. Save enough to rebuild the prompt.

type HallucinationTrace = {
  requestId: string;
  promptVersion: string;
  model: string;
  temperature: number;
  retrieved: Array<{ id: string; score: number; path: string }>;
  packedIds: string[];
  output: string;
  claims: Claim[];
  audit: string[];
};

export function record(t: HallucinationTrace) {
  // Redact PII before this hits the log store.
  return logs.put(t.requestId, t);
}

When a report comes in:

  1. Open the trace. Confirm packedIds (not “we have an index”).
  2. Run auditClaims. Unknown ids mean the model cited fiction. Empty chunkIds means it never tried.
  3. If ids are real, read those chunks. If the claim is not there, it is a generation bug (pressure or sampler). If the claim is there but false in the world, it is a corpus bug.
  4. Hit deadUrls on anything the UI rendered as a link.
  5. Add a golden case. If you only hot-fix the prompt, the next intern will revert it.

Production traces are how you grow the set. See tracing for the span shape.

Footguns

  • Rendering model-authored links as chips. Style is trust. Verify or do not render.
  • Asking the same model “did you hallucinate?” as the only judge. It will often defend itself. Use a rubric and spot-check (LLM as judge).
  • Fixing one famous failure with a new system-prompt sentence. You hid one shape of the bug.
  • High k with no rerank. Extra junk chunks give the model more ways to mix a remix.
  • Tools that return prose the model then “summarizes” into a fake id. Keep tool results structured. Your code owns ids.
  • Prompt injection in a retrieved page that says “the refund window is 365 days.” That is not a creative hallucination. It is an attacker using your retriever. Prompt injection is the sibling.

Interview answer (out loud)

“A hallucination is a fluent claim I cannot support from retrieved text, tool results, or a verified URL. I classify it as a retrieval miss, prompt pressure, sampler noise, or a missing refusal path. I force quote-then-answer with ids I issued, then verify those ids and any links in code. Empty retrieval returns a canned refusal. I debug from a trace that includes packed chunk ids and the prompt version, and I add the case to the golden set instead of only editing the system prompt.”

Further reading (primary sources)

Related guides