Frontend Beauty
ESC

Type to search the knowledge base.

RAG Evaluation Metrics

Split retrieval metrics from answer metrics. Hit rate, MRR, faithfulness, and why a single 1–5 "quality" score hides the broken half.

intermediate6 min read
  • rag
  • evals
  • metrics

You changed the system prompt. The chatbot “feels better.” Support still gets the same wrong refund window, and you only notice after a week because your dashboard is a 1-5 “quality” average. RAG evaluation is two scoreboards. Mixing them is how you ship a friendlier liar.

Retrieval can get worse while the prose gets warmer. The model is good at filling holes. If you only grade the final paragraph, you will reward hole-filling.

Key takeaways

  • Score retrieval and answers on different metrics. Do not average them into one vibe.
  • Hit rate and MRR tell you whether a relevant chunk made the prompt.
  • Faithfulness asks if claims are supported. Helpfulness asks if the user was served.
  • You need labeled queries with relevant chunk ids before you tune prompts.
  • A single 1-5 quality score will hide a retrieval regression that a nicer prompt papered over.

RAG evaluation is two scoreboards

A RAG request fails in layers. Grade the layer you changed.

Layer If this is broken Typical symptom Metric family
Query Follow-up pronouns never rewritten Retrieval for “the other plan” returns nothing useful Rewrite win rate (separate eval)
Retrieval Right doc never in top-k Fluent answer from the wrong policy Hit rate, recall@k, MRR
Ranking Right doc is #17, prompt has 4 slots Almost, then a confident miss MRR, nDCG, precision@k
Generation Right chunks, wrong sentence Citation looks fine, fact is not Faithfulness, exact checks
Product True, useless “Yes, we have a policy.” Helpfulness / task success

Libraries such as Ragas name several of these (context recall, context precision, faithfulness, response relevancy). Use the names if they match your pipeline. Do not import a dozen LLM-judge scores on day one. Start with labels you can defend.

Retrieval first: hit rate, recall, MRR

You need a set of queries, each with one or more relevant chunk ids. Those ids come from humans (or from production traces you reviewed), not from the model you are grading.

type LabeledQuery = {
  id: string;
  query: string;
  relevantChunkIds: string[];
};

/** True if any gold chunk appears in the top-k retrieved ids. */
export function hitRateAtK(
  retrieved: string[][],
  labels: LabeledQuery[],
  k: number,
): number {
  if (labels.length === 0) return 0;
  let hits = 0;
  for (let i = 0; i < labels.length; i++) {
    const top = new Set(retrieved[i]?.slice(0, k) ?? []);
    const hit = labels[i].relevantChunkIds.some((id) => top.has(id));
    if (hit) hits += 1;
  }
  return hits / labels.length;
}

/** Mean reciprocal rank of the first relevant chunk. 0 if none. */
export function meanReciprocalRank(
  retrieved: string[][],
  labels: LabeledQuery[],
): number {
  if (labels.length === 0) return 0;
  let sum = 0;
  for (let i = 0; i < labels.length; i++) {
    const relevant = new Set(labels[i].relevantChunkIds);
    const rank = (retrieved[i] ?? []).findIndex((id) => relevant.has(id));
    sum += rank === -1 ? 0 : 1 / (rank + 1);
  }
  return sum / labels.length;
}

How to read them:

  • Hit rate@k (sometimes called recall@k when one relevant doc is enough): did we even have a chance? If this drops after a chunker change, stop rewriting the prompt.
  • MRR: where did the first good chunk sit? Moving a gold chunk from #1 to #8 with k=4 is a silent miss. Hit rate@10 can look healthy while the prompt is starved.
  • Recall@k with multiple gold chunks: fraction of gold ids in the top-k. Use it when the answer needs two sections (eligibility and the exception).
  • nDCG grades the whole ranking. Fine once you have graded relevance levels. Do not invent a target nDCG from a blog post. Compute yours, then watch deltas.

These metrics are deterministic if retrieval is. ANN search can shuffle ties; pin seeds or accept a small flake budget the same way you would for a timing test.

Answer second: faithfulness vs helpfulness

Once the prompt contains the right text, grade the words.

Faithfulness (Ragas’ definition is the useful one): break the answer into claims, check each claim against retrieved context, score the fraction that are supported. An answer can be faithful and still useless (“The handbook mentions refunds.”). It can also be helpful and unfaithful (correct-sounding 90-day window that is not in the chunk).

Helpfulness / answer relevancy: did we address the question. This is closer to a product rubric. A short “I don’t know, here is the closest section” can outscore a long tour of the wrong page.

Do not use an LLM judge for arithmetic or ids. If the user asked for orderId, assert the string. If they asked whether plan B includes SSO, that is a labeled yes/no. Save the judge for prose. See LLM as judge for position bias and verbosity bias; a judge that loves long answers will fight faithfulness.

type AnswerCase = {
  query: string;
  retrievedTexts: string[];
  answer: string;
  mustInclude?: string[];
  mustNotInclude?: string[];
};

export function assertionScore(c: AnswerCase): { pass: boolean; misses: string[] } {
  const misses: string[] = [];
  for (const needle of c.mustInclude ?? []) {
    if (!c.answer.includes(needle)) misses.push(`missing:${needle}`);
  }
  for (const banned of c.mustNotInclude ?? []) {
    if (c.answer.includes(banned)) misses.push(`banned:${banned}`);
  }
  return { pass: misses.length === 0, misses };
}

Run assertions on every prompt change. Add a faithfulness judge on a sample. Do not replace the assertions with the judge.

A labeled query set you actually own

A golden dataset for RAG is not twelve questions the team wrote from memory. Pull production queries. Cover:

  • Exact titles (“Refund policy”)
  • Paraphrases (“how do I get my money back”)
  • Identifiers (SKU, error code, ORD-18422) that should hit keyword search
  • Follow-ups that need history
  • Adversarial pages: two products with similar names

Each row needs relevantChunkIds and an acceptable answer shape (exact strings, or a short rubric). Version the set. When you re-chunk, remap ids or the retrieval numbers become fiction.

What a 1-5 “quality” score hides

A single scalar will move when you:

  • Make the bot more polite (helpfulness up, faithfulness flat or down)
  • Stuff more chunks (helpfulness up, retrieval precision down, cost up)
  • Switch models (prose quality up, citation discipline down)

Report a small table per release: hit@5, MRR, assertion pass rate, faithfulness on the judged slice, p95 latency, tokens per request. If hit@5 dropped and the 1-5 rose, you optimized the wrong layer. That is the whole point of LLM evals as a product practice: you cannot unit-test a poem, but you can unit-test “did chunk refund-30-day land in the prompt.”

Interview answer (out loud)

“I split RAG evaluation into retrieval and answer. Retrieval is hit rate at the k we actually pack, plus MRR so I know if the gold chunk is first or fourteenth. Answers get deterministic assertions on ids and numbers, then a faithfulness score for claims against the retrieved text, then a separate helpfulness rubric. I would not ship a single 1-5 quality number, because a nicer prompt can hide a worse index. Labels are query plus relevant chunk ids I own, not the model’s self-grade.”

Further reading (primary sources)

Related guides