Frontend Beauty
ESC

Type to search the knowledge base.

LLM Evals

An eval is a scored dataset you run on every prompt change. Offline, sampled online, and why "we tried it and it looked good" is not a release bar.

intermediate5 min read
  • evals
  • quality
  • testing

You cannot unit-test a poem. You can unit-test “did we refund the right order id.” LLM evaluation is that test, run as a scored dataset on every prompt, model, or tool change. “We tried it and it looked good” is a demo. It is not a release bar.

If you already ship CI for React, this is the same habit with worse determinism. You pin inputs, you score outputs, you fail the build when the score drops. Start with assertions on structured fields. Add rubrics only when language is the product.

Key takeaways

  • An eval is a dataset plus a scorer you rerun. A hallway QA pass is not an eval.
  • Exact-match on ids, tools, and JSON first. Rubrics and pairwise later.
  • Offline golden sets catch regressions. Sampled production traces catch drift.
  • Pin grader versions. A moving judge is not a test.
  • Score the field you would argue about in an incident, not a 1-5 “quality” blob.

LLM evaluation is a dataset plus a scorer

OpenAI’s eval docs describe the same two ingredients: test data, and testing criteria (graders). Anthropic’s eval tooling is the same idea in a different console: success criteria, then a set you can rerun. You can also do it in 80 lines in CI. The product is the habit, not the vendor tab.

import { z } from 'zod';

const Output = z.object({
  orderId: z.string(),
  action: z.enum(['refund', 'deny', 'ask_human']),
  amountCents: z.number().int().nonnegative().optional(),
});

type Case = {
  id: string;
  input: { user: string; orderId: string };
  expect: z.infer<typeof Output>;
};

type Score = { caseId: string; pass: boolean; checks: Record<string, boolean> };

export async function runEvals(
  cases: Case[],
  generate: (input: Case['input']) => Promise<unknown>,
): Promise<{ passed: number; scores: Score[] }> {
  const scores: Score[] = [];
  for (const c of cases) {
    const parsed = Output.safeParse(await generate(c.input));
    if (!parsed.success) {
      scores.push({ caseId: c.id, pass: false, checks: { schema: false } });
      continue;
    }
    const checks = {
      schema: true,
      orderId: parsed.data.orderId === c.expect.orderId,
      action: parsed.data.action === c.expect.action,
      amount:
        c.expect.amountCents === undefined ||
        parsed.data.amountCents === c.expect.amountCents,
    };
    scores.push({ caseId: c.id, pass: Object.values(checks).every(Boolean), checks });
  }
  return { passed: scores.filter((s) => s.pass).length, scores };
}

That is an eval: labeled inputs, acceptable outputs, a scorer that does not care how eloquent the model was. A poem about the refund can still fail orderId.

Which scorer?

Scorer Use for Do not use for
Schema / exact match Ids, enums, tool names, JSON fields Tone, summaries
String contains / regex Required citations, refusal phrases Paraphrases
Execution “SQL runs”, “tests still pass” Prose quality
Rubric (LLM judge) Helpfulness when humans defined the rubric Math, amounts, authz
Pairwise A vs B on the same case A release gate with no human calibration

Start at the top of the table. LLM as judge is a later tool: biased toward long polite answers unless you fight it. Golden datasets are how you stop cheating; if the team wrote twelve examples from memory, you measured imagination.

For agents, score the trace as well as the final text: did it call get_order before refund_order? Did it call a forbidden tool? OpenAI’s agent-eval guidance splits tool selection from argument precision. That split is worth copying even if you never use their dashboard.

Offline, sampled online

Offline. A held-out set in git. Run on every prompt PR. This is the eval gate. Keep it small enough to finish in CI, hard enough that a prompt typo fails. Quarantine flaky cases; do not lower the threshold until everything is green.

Online. Sample production traces (with PII redaction). Score them with the same assertions where you can (parsed JSON, tool names). Use a judge only on a slice a human still spot-checks. Online is how you find the query your golden set never dreamed of: slang, new SKUs, empty retrieval.

import { z } from 'zod';

const OnlineOutput = z.object({
  orderId: z.string(),
  action: z.enum(['refund', 'deny', 'ask_human']),
});

type SampledTrace = {
  id: string;
  output: unknown;
  tools: string[];
};

export function scoreOnlineSample(
  traces: SampledTrace[],
  everyNth: number,
  allowedTools: Set<string>,
) {
  return traces
    .filter((_, i) => i % everyNth === 0)
    .map((t) => {
      const parsed = OnlineOutput.safeParse(t.output);
      const checks = {
        schema: parsed.success,
        toolsAllowed: t.tools.every((name) => allowedTools.has(name)),
      };
      return { traceId: t.id, pass: Object.values(checks).every(Boolean), checks };
    });
}

// Nightly: scoreOnlineSample(redactedProdTraces, 20, allowed).
// 1/20 of traffic, same assertions as CI. Not a vibe pass on Slack screenshots.

Keep the online schema aligned with the offline runner. If the two scorers drift, you built two products.

Pair them. Offline is regression. Online is coverage. Neither replaces the other. “Looks good on my laptop” is neither.

Pin versions: prompt hash, model id, tool schema version, grader prompt version. If the judge model silently upgrades, your trend line is a lie.

What not to score with vibes

Do not average a 1-5 “quality” into a single dashboard number and call it health. Helpfulness can go up while refund accuracy dies. Split metrics the way you already split Web Vitals: one number per failure you care about.

Do not eval math by asking another model “does this look right?” Execute it, or compare the field.

Do not put the few-shot examples from the prompt into the golden set and then celebrate 100%. That is training leakage. Hold some cases out.

Do not skip the boring assertions because they feel unlike “AI.” They are the ones that survive an incident review.

Interview answer (out loud)

“An LLM eval is a scored dataset I rerun on every prompt or model change. I start with schema and exact-match on structured fields like order id and action, then add rubrics only for language. Offline golden sets are the CI gate. Sampled production traces catch drift. ‘We tried it and it looked good’ is not a bar. I pin grader versions so the test does not move under my feet.”

Further reading (primary sources)

Related guides