Frontend Beauty
ESC

Type to search the knowledge base.

Golden Datasets for LLM Apps

A golden set is labeled inputs and acceptable outputs you own. How to collect it from production, keep it from rotting, and stop cheating on it.

intermediate5 min read
  • evals
  • datasets
  • quality

Twelve examples the team typed from memory will pass any prompt you already like. That is not a golden dataset. That is a mood board. The set only starts working when rows come from production, from the tickets you actually lose, and from the intents you claim to support.

A golden set is labeled inputs plus acceptable outputs you own. You version it. You refuse to tune the prompt on the same rows you use as a ship bar. If you cannot say which slice is held out, you do not have an eval. You have a rehearsal.

Key takeaways

  • Own the labels. A row is an input, constraints, and a checkable expected (fields, refuse, citations), not a pretty paragraph.
  • Mine production traces and real failures first. Invented happy paths hide the product.
  • Cover intents and hard cases, not “more JSON.” Track holes the way you track code coverage.
  • Split few-shot examples from the held-out set. Mixing them is how you cheat.
  • Version the dataset with the prompt. Stale gold is how last quarter’s policy still “passes.”

What a golden dataset is actually for

The set answers one question: did this prompt, retriever, or tool loop get worse? It is not training data. It is not a dump of chat logs. It is a curated, versioned table you can run in CI.

type GoldenRow = {
  id: string;
  intent: string;
  difficulty: 'easy' | 'hard' | 'adversarial';
  input: {
    question: string;
    locale: 'en' | 'es';
    retrievedChunkIds?: string[];
  };
  expected: {
    // Prefer checkable fields over an essay.
    orderId?: string;
    mustCite?: string[];
    refuse?: boolean;
    forbiddenTools?: string[];
    answerContains?: string[];
    answerExcludes?: string[];
  };
  source: 'prod' | 'expert' | 'synthetic';
  addedIn: string; // prompt or app version that introduced the row
};

If the only expected value is a free-text “ideal answer,” you will need a judge. Start with fields you can assert. Refund the right orderId. Refuse when retrieval is empty. Do not call charge_card. Those rows stay cheap forever.

Where rows should come from

Pull from the live system, then fill gaps on purpose.

  1. Failed traces. User thumbs-down, empty retrieval, tool errors, hallucinated ids. This is the highest-value source. See tracing.
  2. Intent catalog. List the jobs the product claims: tracking, returns, cancel, “what is on the invoice.” Each intent needs easy and hard rows.
  3. Expert gold. A support lead writes the acceptable answer for policy questions. Do not let the model draft the label you will score it against.
  4. Synthetic last. Use generation to propose paraphrases, then a human accepts or edits. Unreviewed synthetic rows train the eval to like the generator’s accent.

Tag every row with source and intent. When a score drops, you want “returns / adversarial” in the diff, not a single melted average.

Coverage beats volume

A 40-row set that hits every intent and the nasty cases beats 400 near-duplicates of “where is my package.” Aim for coverage the way you would for QA:

Slice Why it exists Example row
Happy path Prove the default still works Tracking number in, ETA out
Ambiguous follow-up History-dependent queries “what about the other plan?”
Missing evidence Refusal path Question your index cannot answer
Injection / untrusted file Safety Retrieved HTML that asks to email the transcript
Locale / format Tokenizer and prompt drift Same intent, different language
Policy edge Humans disagree here Partial refund after the window

Hard cases are not optional flavor. They are why you have a set. If every row is something a new hire would get right, the eval cannot fail, so it cannot protect you.

Keep a written coverage map (intent × slice). Empty cells are tickets, not “we’ll add later.”

Versioning and the held-out split

Treat the dataset like a fixture file in git, or a versioned table with a changelog.

  • Few-shot pool. Examples you are allowed to paste into the prompt. Small. Reviewed. Not used as the ship bar.
  • Held-out eval. Never appears in the prompt, never used to “tweak the wording until green.”
  • Canary / new prod. Rows added from last week’s traces. They start in a quarantine tag until you decide they are stable.
const HELD_OUT = golden.filter((row) => row.id.startsWith('eval-'));
const FEW_SHOT = golden.filter((row) => row.id.startsWith('shot-'));

function assertNoLeak(heldOut: GoldenRow[], fewShot: GoldenRow[]) {
  const shotQuestions = new Set(fewShot.map((r) => normalize(r.input.question)));
  const leaked = heldOut.filter((r) => shotQuestions.has(normalize(r.input.question)));
  if (leaked.length) {
    throw new Error(`few-shot leaked into held-out: ${leaked.map((r) => r.id).join(',')}`);
  }
}

function normalize(q: string) {
  return q.trim().toLowerCase().replace(/\s+/g, ' ');
}

When policy changes, do not silently edit old labels. Add a new version of the row, retire the old one, and record why. Otherwise you cannot tell a prompt regression from a dataset edit.

How teams cheat on their own set

  • Teaching to the test. You keep adding clauses to the prompt that mention the 12 examples by name. Score goes up. Production does not.
  • Gold written by the model. Self-graded homework. The eval will prefer that model’s style forever.
  • Only the demo path. Sales-script questions, no angry users, no empty index, no “ignore previous instructions.”
  • Unbounded growth. Five thousand unlabeled prod dumps is not gold. It is a haystack you will never relabel.
  • One global score. A routing bug can be hidden by a summarization win. Report by intent.

The antidote is boring: a small held-out set, production sampling, and eval gates in CI that fail on the slices you care about. LLM evals cover scoring. This page is the data those scores sit on.

Interview answer (out loud)

“A golden dataset is a versioned set of real inputs with checkable expected outputs that I own. I collect from production failures and an intent map, keep few-shot examples out of the held-out split, and prefer field assertions over graded essays. I would rather have 60 labeled hard cases than 600 paraphrases of the happy path. If the set is written from memory, I am measuring imagination, not the product.”

Further reading (primary sources)

Related guides