Frontend Beauty
ESC

Type to search the knowledge base.

RAG Interview Questions

RAG interview questions on chunking, hybrid search, ACLs, and evals. Practice answers that name the layer that broke, not a vendor.

intermediate6 min read
  • interview
  • rag

If your answer is “we use a vector DB,” the next question will be “and when does BM25 win.” RAG interview questions are scenario questions. They poke the layer that actually broke: a table split in half, a tenant filter applied in the prompt, a citation that remixes chunk 3 into a policy you never published.

Retrieval-augmented generation is search, pack, generate, cite. The model only sees what you stuffed in. Miss the search and the answer is fan fiction with better grammar. That is the whole mental model. The rest is how you prove it.

Key takeaways

  • Say which stage failed: parse, chunk, index freshness, retrieve, pack, generate, cite.
  • Keyword search still wins on SKUs, error codes, and rare names. Hybrid is the default, not a flourish.
  • ACLs are a pre-filter on the index, never a request that the model “not mention” a doc.
  • Empty retrieval should refuse. A citation id must exist in the packed context.
  • Measure retrieval and answers on different scores or you will tune the prompt while recall dies.

A packing budget you can draw

Interviewers like a table more than a cloud diagram.

Slot What you put there Failure if you skip the budget
System Policy, refusal rule, citation format Model ignores the last sentence of a novel-length prompt
Retrieved chunks Top n after filter + rerank, with ids Lost-in-the-middle, or a 20k dump that buries the hit
History Last turns, or a summary Follow-up “the other plan” searches the wrong string
User The actual question Query rewriting never ran, index never saw the SKU
Generation Reserved max tokens Cut-off mid-citation 
type Chunk = { id: string; url: string; text: string; tenantId: string };

function packContext(args: {
  allowedTenant: string;
  hits: Chunk[];
  tokenBudget: number;
  countTokens: (s: string) => number;
}): Chunk[] {
  const scoped = args.hits.filter((h) => h.tenantId === args.allowedTenant);
  const packed: Chunk[] = [];
  let used = 0;
  for (const hit of scoped) {
    const cost = args.countTokens(hit.text);
    if (used + cost > args.tokenBudget) break;
    packed.push(hit);
    used += cost;
  }
  return packed;
}

Filter before rank. Counting tokens with string.length is the wrong meter, but in a whiteboard, say you will use the real tokenizer later. Tokens and context windows.

Eleven RAG interview questions

1. What is RAG, in one breath?

Say this: “I search my corpus, pack the hits into the prompt, and ask the model to answer from that text. The weights do not contain my handbook. If retrieval is empty I refuse, I do not let the model improvise policy.”

Common miss: “We store the docs in a vector database so the model remembers them.” The index is not memory. RAG mental model.

2. When does BM25 beat embeddings?

Say this: “SKUs, error codes, order ids, rare proper names. ‘ORD-18422’ is a terrible embedding query and a perfect keyword query. I run hybrid and merge with reciprocal rank fusion, then rerank if the prompt only has room for four chunks.”

Common miss: Claiming vectors made lexical search obsolete. Hybrid search.

3. A table got split across chunks. What happens?

Say this: “The retriever returns half a truth: a header without units, or a row without the column names. I chunk on headings and keep tables together, with overlap as a paid extra, not a default. Metadata keeps url and heading path so citations can point at a real place.”

Common miss: “We use 500-character chunks with 50 overlap” as if that were a principle. Chunking.

4. Production answers quote a price you deprecated last sprint.

Say this: “Stale vectors. Ingestion must upsert by content hash and delete by source. If I cannot rebuild the index from files, I have a snowflake. I would check whether the old chunk is still nearest-neighbor and whether deletes actually tombstoned.”

Common miss: Blaming the model for “not knowing the new price.”

5. The intern asked, and the CEO’s draft appeared in the prompt.

Say this: “Similarity does not know ACLs. Every vector carries tenant and permission metadata. The query applies a pre-filter for what this user can read. I would write an eval that a forbidden doc never appears in the packed context, even as a negative example.”

Common miss: “The system prompt says not to reveal confidential docs.” Permission-aware retrieval.

6. Retrieval returns nothing. What does the UI do?

Say this: “Refuse with a specific empty state: we could not find this in the docs, try X. I do not lower the score threshold until something, anything, comes back. A low-similarity chunk is how you get confident nonsense.”

Common miss: Always generating, then adding fake citations to look grounded.

7. The footnote points at chunk 3. Is that proof?

Say this: “No. The sentence can still be a remix. I force quote-then-answer, verify the citation id was packed, and optionally check entailment with a judge or a string overlap for extractive claims. The UI should let you open the source, not just see a chip.”

Common miss: Treating citation UI as grounding. Citations.

8. Follow-up: “what about the other plan?”

Say this: “The index has never heard of ‘the other plan.’ I rewrite the query with the prior turn into a standalone search string, keep SKUs intact, and accept the extra latency. I do not embed the whole chat history as one query.”

Common miss: Sending the raw follow-up to kNN and shrugging.

9. How do you eval RAG without lying to yourself?

Say this: “Labeled queries with relevant chunk ids. Hit rate / MRR on retrieval, separate from answer faithfulness and helpfulness. If I only score the final paragraph, a prompt change can look better while recall got worse.”

Common miss: One 1-5 “quality” number. RAG evaluation metrics.

Say this: “Parser dumped chrome: nav, cookie banner, related links. I strip boilerplate, keep heading order, and drop repeating templates before embed. PDFs are layout, not text. If reading order is wrong, retrieval is wrong.”

Common miss: “We used a better embedding model.”

11. When is RAG the wrong tool?

Say this: “Live balances, counts, permissions, and anything I should compute. I call a tool or run SQL. RAG is for prose that changes slower than a query planner. Fine-tuning is not a substitute for today’s policy PDF.”

Common miss: Building a slower database out of chunks. When RAG is the wrong tool.

Interview answer (out loud)

“RAG is search then generate, with a token budget. I filter by tenant first, hybrid-search, pack ids, and refuse on empty hits. A citation chip is not entailment. I eval retrieval separately from the answer so I do not tune the prompt while the index rots. If the fact belongs in SQL, I do not embed the catalog.”

Further reading (primary sources)

Related guides