Frontend Beauty
ESC

Type to search the knowledge base.

When RAG Is the Wrong Tool

RAG cannot fix stale facts you should query, math you should compute, or permissions you should enforce in code. Know when to skip the index.

intermediate6 min read
  • rag
  • architecture
  • tradeoffs

You indexed the SQL catalog. The bot now “knows” there is an orders table. A user asks how many invoices are open for their account. The model retrieves a CREATE TABLE snippet and guesses. Congratulations: you built a slower, worse database. That is the moment to learn when not to use RAG.

Retrieval-augmented generation is search, then generation, with a budget. It is the right shape for prose that changes on a human timescale: policies, runbooks, this site’s MDX. It is the wrong shape for live rows, totals, permissions, and anything you can compute without sampling.

Key takeaways

  • An index is not a database, a calculator, or an ACL.
  • Query live systems for counts, balances, and status. Do not embed yesterday’s dump.
  • Do arithmetic and aggregations in code. Retrieve the policy that explains the number.
  • Permissions belong in the handler. Similarity does not know tenancy.
  • Tools, SQL, a short prompt, or a fine-tune each beat RAG for jobs they actually own.

Should this query hit an index?

Walk the question through a cheap filter before you spend an embedding call.

type Route =
  | { kind: 'sql'; intent: 'invoice_balance' | 'open_ticket_count' }
  | { kind: 'tool'; name: 'refund_order' | 'reset_password' }
  | { kind: 'compute'; expr: 'prorate' | 'tax' }
  | { kind: 'prompt'; topic: 'tone' | 'output_shape' }
  | { kind: 'retrieve'; corpus: 'docs' | 'runbooks' };

function routeBillingQuestion(q: {
  text: string;
  needsLiveRow: boolean;
  needsWrite: boolean;
  needsArithmetic: boolean;
  isStyleOnly: boolean;
}): Route {
  if (q.needsWrite) return { kind: 'tool', name: 'refund_order' };
  if (q.needsLiveRow) return { kind: 'sql', intent: 'invoice_balance' };
  if (q.needsArithmetic) return { kind: 'compute', expr: 'prorate' };
  if (q.isStyleOnly) return { kind: 'prompt', topic: 'tone' };
  return { kind: 'retrieve', corpus: 'docs' };
}

The classifier can be a small structured-output call. The decision is still yours. If needsLiveRow is true, do not “also retrieve the warehouse glossary” and hope the model picks the right number.

When not to use RAG: four jobs an index cannot do

1. Freshness you should query

Order status, seat count, feature flags, “is this SKU in stock,” “what’s on call right now.” These change faster than you re-embed. Even an hourly dump is a lie with a timestamp. Give the model a read tool (or run SQL in your process) and put the result in the prompt as data, not as a chunk that might be from Tuesday.

A catalog comment that says orders.status is enum('open','paid','void') is documentation. It is not the row.

2. Math you should compute

Proration, tax, “how many of these match,” date windows. Language models are samplers. They will produce a plausible total. Your billing service already has the function. Call it. If you also need to explain the number, retrieve the policy paragraph and attach the computed result as a tool observation.

If you catch yourself chunking spreadsheets so the model can “add them up,” stop. That is a job for code, then maybe a one-line summary.

3. Permissions you should enforce

Vector search does not know GDPR. If the intern’s query is close to the CEO’s draft, cosine will fetch it. Filtering in the prompt (“please ignore other tenants”) is not a control. Filter in the query with tenant metadata, or do not put the doc in that user’s candidate set. This is permission-aware retrieval when you do retrieve. When the object is a row in Postgres, skip retrieval and run the same WHERE user_id = $1 you already trust.

4. Deterministic workflows

Password reset, refund, “add this user to the project.” Those are tools with auth, idempotency, and often a human pause. An indexed help article that describes the click path is useful after you decide the user wanted an explanation. It is a bad substitute for function calling.

Tools vs retrieval vs a prompt vs weights

Job Reach for RAG fails because
“What’s our refund window?” Retrieve the policy page This is the happy path
“What’s my refund status?” SQL / order API The policy page has no row
“Refund order 18422” Tool + auth + approval Side effects are not search
“How much is 17% of $84.10?” compute The model will round creatively
“Always answer in our JSON shape” Schema / structured output; fine-tune if stubborn Format is not a document
“Sound like our brand” System prompt; fine-tune if still off Tone is not in the vector index
Policy changed this morning Re-index or a tool that reads the CMS Stale chunks win until you refresh

Fine-tuning does not rescue a freshness problem. Weights will learn last quarter’s policy. The comparison lives in RAG vs fine-tuning. The short version: retrieve facts, tune format.

A short, stable policy (three bullets that fit in the system prompt) does not need an index. Putting it in the prompt is cheaper, cacheable, and easier to version. Index the long tail.

What “we still have RAG” should mean

Plenty of products mix paths in one chat turn:

  1. Classify the intent with structured output.
  2. Run SQL or a tool for the live bit.
  3. Retrieve a paragraph if the user also asked “why.”
  4. Generate a reply that cites the paragraph and inserts the number from the tool, not from the model’s head.

The failure mode is skipping step 2 because step 3 “might have a table.” Parsed HTML tables in a chunk go stale the same day finance updates Stripe. If you need the table, query it.

On the frontend this looks like any other BFF: a route handler that is allowed to hit the database, and a model that is allowed to propose which handler. The model does not get a connection string.

Interview answer (out loud)

“When not to use RAG: live facts I should query, math I should compute, permissions I should enforce in code, and writes that belong in a tool. RAG is for prose I do not want in the system prompt, with citations. I would classify the intent, run SQL or a function for rows and totals, and retrieve only the explanation. Indexing a schema catalog instead of running SQL is a slower, worse database. Fine-tuning does not fix freshness.”

Further reading (primary sources)

Related guides