Frontend Beauty
ESC

Type to search the knowledge base.

Eval Gates in CI

Fail the build when a golden set regresses. Thresholds, flake budgets, and why an LLM judge in CI needs pinning and retries of its own.

advanced6 min read
  • evals
  • ci
  • llmops

A prompt pull request that drops refund-id accuracy by eight points should be as blocked as a red unit test. If the only review is “I tried it in the playground,” you are shipping folklore. Eval gates in CI mean a golden set runs on the change, a threshold fails the job, and the merge is not a vibe.

Judges flake. Models move. CI minutes are not free. Put deterministic checks first, pin versions, give the judge a small retry budget, and quarantine cases that disagree with themselves.

Key takeaways

  • Gate on assertions you can replay: schema, exact ids, allowlists, retrieval hit rate.
  • LLM-as-judge is a second job with its own pin, timeout, and flake budget.
  • Fail closed on product-critical slices. Do not average them away in a single 1–5 score.
  • Pin prompt version, model snapshot, and dataset git sha in the report.
  • Quarantine flaky cases. A noisy gate that people skip is worse than no gate.

What belongs in an eval gates CI job

Not every quality idea belongs on pull_request. CI needs a short, stable, labeled set. Long-tail online evals live after merge.

Layer Example Flake risk Put in PR gate?
Schema / types Zod parse, required keys Low Yes
Exact / set match orderId, enum labels Low Yes
Retrieval Gold chunk in top-k Low if index is fixture Yes
Programmatic rules Totals sum, URL allowlist Low Yes
Rubric judge “Helpful, grounded” High Sample, or nightly
Human Brand voice n/a Not CI

If you cannot write the assertion without a paragraph of taste, it is not a gate yet. It is a dashboard.

The dataset is a versioned artifact: inputs, expected fields, and tags (refund, empty-retrieval, injection). How you collect it is golden datasets. How you think about scorers is LLM evals. This page is the part that fails the build.

A workflow that fails closed

Keep the runner boring. A Node script, a fixture index, secrets from the environment. GitHub Actions is fine.

# .github/workflows/llm-eval.yml
name: llm-eval
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'evals/**'
      - 'src/lib/llm/**'
      - '.github/workflows/llm-eval.yml'
jobs:
  gate:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
          cache: npm
      - run: npm ci
      - run: npm run eval:gate
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          EVAL_MODEL: gpt-4.1-mini-2025-04-14

The script should exit non-zero. Do not parse a markdown summary with a human in the loop and then click merge.

import { z } from 'zod';

const Extract = z.object({
  orderId: z.string(),
  refundable: z.boolean(),
  citationIds: z.array(z.string()),
});

type Case = {
  id: string;
  tags: string[];
  input: { question: string; orderId: string };
  expect: { orderId: string; refundable: boolean; mustCite?: string };
};

export async function runGate(cases: Case[], run: (c: Case) => Promise<unknown>) {
  const failures: string[] = [];

  for (const c of cases) {
    let parsed: z.infer<typeof Extract>;
    try {
      parsed = Extract.parse(await run(c));
    } catch {
      failures.push(`${c.id}: schema`);
      continue;
    }
    if (parsed.orderId !== c.expect.orderId) failures.push(`${c.id}: orderId`);
    if (parsed.refundable !== c.expect.refundable) failures.push(`${c.id}: refundable`);
    if (c.expect.mustCite && !parsed.citationIds.includes(c.expect.mustCite)) {
      failures.push(`${c.id}: citation`);
    }
  }

  const bySlice = (tag: string) => {
    const slice = cases.filter((c) => c.tags.includes(tag));
    const failed = failures.filter((f) => slice.some((c) => f.startsWith(c.id))).length;
    return slice.length === 0 ? 1 : 1 - failed / slice.length;
  };

  const overall = 1 - failures.length / cases.length;
  const refund = bySlice('refund');

  if (refund < 0.95) throw new Error(`refund slice ${refund.toFixed(3)} < 0.95`);
  if (overall < 0.9) throw new Error(`overall ${overall.toFixed(3)} < 0.90`);

  console.log(JSON.stringify({ overall, refund, failures }, null, 2));
}

Slice thresholds matter. A 99% overall score that hides a 70% refund slice is how you merge a polite, wrong bot.

Flakes, judges, and pinned models

Deterministic checks should not need retries. Network 503s on the candidate model can retry with the same policy as production (retries, timeouts, fallbacks), inside the job timeout.

Judges are different:

  • Pin the judge snapshot. gpt-latest in CI is how Friday’s pass becomes Monday’s fail with no prompt change.
  • Pin the candidate. Evaluating a moving alias is not a regression test.
  • Blind the order if you pairwise-compare. Position bias is real.
  • Retry the judge, not the product. If two judge calls disagree, mark the case flaky and do not fail the whole gate on one coin flip unless the slice is critical and both calls failed the rubric.
  • Budget. A 400-case judge on every docs typo PR will be disabled by the first person who pays the bill. Keep PR gates small; run the long set on main or nightly.

Quarantine is a file, not a Slack message:

type Quarantine = { id: string; reason: string; added: string };

export function loadCases(all: Case[], q: Quarantine[]) {
  const skip = new Set(q.map((x) => x.id));
  const skipped = all.filter((c) => skip.has(c.id));
  if (skipped.length / all.length > 0.05) {
    throw new Error('quarantine exceeds 5%; fix cases or the judge');
  }
  return all.filter((c) => !skip.has(c.id));
}

A quarantine that grows forever is a skipped test suite. Cap it, and make adding an id a reviewed line in git.

Cost, secrets, and the rest of LLMOps

CI uses a real key. That key is a secret, not an org-wide playground token with production quota. Scope it, cap spend, and do not print prompts that contain customer fixtures into GitHub logs.

Cache embeddings for a frozen retrieval fixture so you are not re-embedding the handbook on every PR. Do not cache the candidate prompt result across commits: that is the thing you are testing.

When the gate is green, still canary in production. CI is the cheap filter. LLMOps is the rest: traces, online sampling, dataset refresh. A gate that never sees production tickets will rot.

Provider outages will fail your PRs. That is painful and correct if the product cannot ship without that provider. If you must merge docs-only changes during an outage, path-filter the workflow so README edits do not call the model.

Interview answer (out loud)

“An eval gate is a CI job that runs a versioned golden set and fails the build on slice thresholds, not a playground screenshot. I put schema and exact-match assertions on the PR path, pin both the candidate and the judge, and keep the LLM rubric small enough to afford. Flaky judge cases get quarantined in git with a cap. Refund-like slices have their own threshold so a high average cannot hide them. The job prints a report with prompt version, model snapshot, and dataset sha.”

Further reading (primary sources)

Related guides