Frontend Beauty
ESC

Type to search the knowledge base.

RAG vs Fine-Tuning

RAG supplies facts at request time. Fine-tuning changes weights. Most product problems are RAG or tools. Fine-tune for stubborn format and tone.

intermediate6 min read
  • rag
  • fine-tuning
  • architecture

Finance posted a new refund policy at 09:12. The chatbot is still quoting last quarter. That is not a “smarter model” ticket. RAG vs fine-tuning is the question of whether the answer should come from documents you can update this afternoon or from weights you can retrain later.

Retrieval-augmented generation stuffs evidence into the prompt at request time. Fine-tuning changes next-token probabilities. Most product bugs are missing evidence, a stale index, or a tool you should have called. Fine-tune when the model has the facts and still will not behave.

Key takeaways

  • RAG (and tools) supply facts, citations, and tenant data. Fine-tuning supplies habit: format, tone, routing.
  • Freshness and attribution almost always vote RAG. Style at high volume can vote fine-tune.
  • You can combine them: retrieve, then generate with a tuned small model. Do not mix the stories in the design doc.
  • If the truth lives in SQL, call SQL. Neither RAG nor fine-tuning is a database.
  • Pick with an eval, not with a preference for “we trained something.”

RAG vs fine-tuning is a product decision

The table is the page. Freshness, citations, and tenant data vote retrieve. Stubborn format and tone vote a weight update, and only after a prompt plus structured outputs failed an eval. Live numbers vote a tool: a route handler that queries the order service, not an index of yesterday’s orders table.

When RAG is the wrong tool is the “call SQL” argument. When fine-tuning helps is the data bar for the other column. This page is which column you are in, and how you score a mix.

Decision table

Need RAG Fine-tune Tool / code
Policy that changes this week Yes No Maybe a CMS read
Cite the paragraph you used Yes No n/a
Per-tenant private docs Yes, with ACL filters Risky (data in weights) n/a
JSON / ticket schema the model keeps breaking Supporting Yes, if prompt failed Structured outputs first
Brand voice on generic chat Optional style snippets Sometimes Prompt
“What is order 18422’s status?” No No Query the order service
Math, totals, tax No No Compute
High QPS, same format, tiny model After retrieval Distill the format n/a

Read the table as votes, not as a personality test. Freshness, attribution, and private data are usually enough to reject a fine-tune. Format and tone are usually enough to reject a bigger index.

A policy change in code, not in weights

The product move is: documents are the source of truth, retrieval is the loader, the model is the speaker.

type PolicyChunk = {
  id: string;
  path: string;
  text: string;
  updatedAt: string;
};

async function answerRefundQuestion(q: {
  question: string;
  tenantId: string;
}): Promise<{ text: string; citations: string[] } | { refuse: true }> {
  const chunks = await retrieve(q.question, {
    tenantId: q.tenantId,
    prefix: 'docs/policies/',
    k: 6,
  });

  if (chunks.length === 0) {
    return { refuse: true };
  }

  const packed = chunks
    .map((c) => `[${c.id} updated ${c.updatedAt}]\n${c.text}`)
    .join('\n\n');

  const completion = await generate({
    // Base or tuned: either still only sees `packed`.
    system:
      'Answer only from the policy chunks. Quote ids. If they do not cover the question, refuse.',
    messages: [
      { role: 'user', content: `Question: ${q.question}\n\nChunks:\n${packed}` },
    ],
  });

  return {
    text: completion.text,
    citations: chunks.map((c) => c.path),
  };
}

// Fine-tuning analog (do not do this for a living policy):
// train on { question, "14-day refund" } rows. Tomorrow finance
// switches to 30 days and the weights keep saying 14.

Re-index is an hour-scale ops problem. Retrain is a dataset-and-eval problem. If the answer must mention this morning’s number, you already know which column of the table you are in. A 09:12 policy file is a retrieval freshness test. It is not a reason to collect new SFT rows.

Combining them without mixing the stories

A tuned model can still read retrieved text. That is a normal production shape:

  1. Retrieve the right chunks (RAG owns recall and ACLs).
  2. Generate with a small fine-tuned model that emits your citation JSON without a three-page system prompt.

What you must not do is explain a factual miss as “we need more epochs.” If the chunk never entered the prompt, more epochs will not fetch it. Split the eval so a format win cannot hide a search bug:

type ComboCase = {
  id: string;
  goldChunkId: string;
  retrievedIds: string[];
  answer: { citationIds: string[]; jsonOk: boolean; usedOnlyRetrieved: boolean };
};

export function scoreCombo(c: ComboCase) {
  const retrieval = c.retrievedIds.includes(c.goldChunkId);
  const faithfulness =
    c.answer.usedOnlyRetrieved &&
    c.answer.citationIds.every((id) => c.retrievedIds.includes(id));
  const format = c.answer.jsonOk;
  return {
    retrieval,
    faithfulness,
    format,
    pass: retrieval && faithfulness && format,
  };
}
Slice Question Owner
Retrieval Did the gold chunk appear in top-k? Index, chunking, filters
Faithfulness Did the answer stay inside those chunks? Prompt, citations, verifier
Format Did the JSON parse and pass business rules? Schema, maybe a fine-tune

Fine-tuning can lift format while retrieval and faithfulness stay flat or get worse (the model becomes more fluent at ignoring evidence). Report the three numbers. A single 1-5 “quality” will celebrate the wrong layer. Citations are how faithfulness shows up in the UI.

The wrong reasons to pick either

“Fine-tune so it knows our company.” Weights are a blurry, delayed, uncited copy of your wiki. Put the wiki in the index.

“RAG so we never prompt-engineer.” Garbage chunks in, confident summaries out. You still write a prompt. You also write chunking and filters.

“We have PDFs, therefore RAG.” If the question is “sum this column,” run a parser and arithmetic. If the PDF is a scan, you have an OCR problem before you have an LLM problem.

“We have 40 examples, therefore fine-tune.” You have few-shot material. Put it in the prompt until the eval saturates.

“Bigger context window, therefore dump the repo.” That is not RAG and not fine-tuning. It is a bill. Retrieval still wins when the needle is one file in a haystack.

Private data in a fine-tune has a long tail: the model can regurgitate training rows, you cannot ACL a weight, and deleting a tenant means another job. Retrieval plus permission-aware filters is the control that actually maps onto tenants.

Interview answer (out loud)

“RAG puts current, citable, tenant-scoped text into the prompt. Fine-tuning changes weights so format and tone are more stable. If the policy can change this week or I need a footnote, I retrieve. If the model already sees the right context and still breaks the schema, I consider a fine-tune after the prompt and structured outputs fail the eval. Live numbers go through tools. I score retrieval, faithfulness, and format separately so I do not train my way out of a search bug.”

Further reading (primary sources)

Related guides