Frontend Beauty
ESC

Type to search the knowledge base.

RAG Mental Model

Retrieval-augmented generation is search, then generation, with a budget. The model only sees what you stuffed into the prompt.

beginner6 min read
  • rag
  • retrieval

The model did not “know” your handbook. You searched, you pasted, it summarized. Miss the search and the answer is fan fiction with a confident tone.

Retrieval augmented generation is that loop with a name: find text, pack it into the prompt, ask the model to answer from that, preferably with citations. The weights stay frozen. The model’s working memory is whatever survived your budget.

Key takeaways

  • RAG is search, then pack, then generate. The model only sees the packed prompt.
  • A vector database is one retriever. BM25, SQL, and grep can play the same role.
  • Failures live in stages: bad index, bad retrieve, bad pack, bad generate, fake cite.
  • Empty retrieval should refuse. Stuffing noise is how you get fluent wrong answers.
  • Cite only ids the model named that you actually packed. Intersect, then drop the rest.

Retrieval augmented generation is a pipeline

Five stages. Each one can lie independently.

  1. Index. Split documents into chunks, attach metadata (url, heading path, tenant, date), store text plus whatever you search with (embeddings, keywords, both).
  2. Retrieve. Turn the user question into a query. Return top-k chunks.
  3. Pack. Fit a subset into the context window with instructions: answer from sources, or say you cannot.
  4. Generate. Sample an answer. Same request, different Tuesday.
  5. Cite. Attach chunk ids the UI can open. Verify the ids exist in this packed set before you render them as links.

If you only remember one sentence: the generator is not looking at your corpus. It is looking at a string you built.

Where it breaks (by stage)

Stage Typical failure What you see
Index Split a table in half, skip a PDF column, stale doc Confident half-truths
Retrieve Wrong query, no keywords, no tenant filter Neighboring-but-wrong chunks
Pack Truncate the useful paragraph, dump 40 near-duplicates Lost-in-the-middle, wasted tokens
Generate Sampler invents a number the chunks never stated Fluent lie
Cite Model names chunk-3 that was not in the prompt Clickable 404

Do not recap chunking or similarity search here. This page is the wiring diagram. Those pages are the knobs.

RAG is not a vector database

People say “we added RAG” and mean “we stood up Pinecone.” That is like saying “we added search” and meaning “we created an Elasticsearch cluster.”

You can retrieve with:

  • Keyword ranking (BM25) for SKUs, error codes, rare names
  • Vector nearest-neighbor for paraphrase
  • Structured lookup (SQL, CMS API) when the fact is a row, not a paragraph
  • A mix, then a rerank

The generation step does not care which retriever you used. It cares whether the packed text contains the answer, and whether you told it to stay inside that text.

When the right move is a tool (get_order(id)), not an index, skip RAG. When RAG is the wrong tool is the sibling argument.

Pack, then cite only what was packed

Retrieved chunks share the context window with the system prompt, history, and reserved output tokens. Count that budget in tokens (tokens and context windows); pack() below uses string.length so the control flow is visible.

type Chunk = {
  id: string;
  text: string;
  sourceUrl: string;
  headingPath: string[];
  score: number;
};

type Completion = {
  text: string;
  // Model-emitted ids. Not trusted until they intersect `packed`.
  citedIds: string[];
};

function pack(chunks: Chunk[], maxChars: number): Chunk[] {
  const chosen: Chunk[] = [];
  let used = 0;
  for (const c of chunks) {
    if (used + c.text.length > maxChars) break;
    chosen.push(c);
    used += c.text.length;
  }
  return chosen;
}

function formatSources(chunks: Chunk[]): string {
  return chunks
    .map(
      (c) =>
        `<source id="${c.id}" url="${c.sourceUrl}">\n${c.text}\n</source>`,
    )
    .join('\n\n');
}

function citationsUsed(citedIds: string[], packed: Chunk[]): string[] {
  const allowed = new Set(packed.map((c) => c.id));
  const seen = new Set<string>();
  const out: string[] = [];
  for (const id of citedIds) {
    if (!allowed.has(id) || seen.has(id)) continue; // drop fake or duplicate ids
    seen.add(id);
    out.push(id);
  }
  return out;
}

async function answerFromRag(
  query: string,
  retrieve: (q: string) => Promise<Chunk[]>,
  generate: (input: { messages: unknown[] }) => Promise<Completion>,
) {
  const hits = await retrieve(query);
  const packed = pack(hits, 12_000);
  if (packed.length === 0) {
    return { text: 'I do not have sources for that.', citations: [] as string[] };
  }

  const completion = await generate({
    messages: [
      {
        role: 'system',
        content:
          'Answer only from <source> blocks. If they do not contain the answer, say so. Return citedIds for sources you actually used, never ids that were not provided.',
      },
      {
        role: 'user',
        content: `${formatSources(packed)}\n\nQuestion: ${query}`,
      },
    ],
  });

  return {
    text: completion.text,
    citations: citationsUsed(completion.citedIds, packed),
  };
}

Failure mode in one comment: if retrieve returns chunks about a different product with a similar name, generate will summarize that product. The model did its job. Your search did not.

A second failure, the one the table calls a clickable 404: attaching packed.map((c) => c.id) teaches the UI that every stuffed chunk was used. It was not. The model may name chunk-3 that was never packed, or skip a packed id that it never quoted. Intersect. Citations and grounding is the entailment check on top of that allow-list.

What the UI owes the user

A spinner that becomes a paragraph is a chatbot. RAG as a product usually needs:

  • Citations the user can open. A footnote that does not match the sentence is worse than no footnote.
  • A refusal path. “No sources” is a feature. Inventing policy is an incident.
  • Trace ids. You will debug “it was weird” only if you stored query, packed chunk ids, model-emitted ids, prompt version, and token counts.

Untrusted retrieved HTML is still untrusted text. Treat it like XSS: it can carry instructions, not only facts.

Interview answer (out loud)

“RAG is search plus generation. I index chunks with metadata, retrieve top-k for the question, pack a budgeted subset into the prompt, and ask the model to answer only from those sources. The model never sees the rest of the corpus. A vector database is optional. Failures are usually retrieval or packing, not ‘the LLM forgot.’ Empty hits should refuse. Citations are the ids the model named, intersected with the packed set; anything else is a fake cite.”

Further reading (primary sources)

Related guides