Frontend Beauty
ESC

Type to search the knowledge base.

Query Rewriting for Retrieval

Users type pronouns and slang. Query rewriting expands, decomposes, or HyDE-drafts a better search string before you hit the index.

intermediate5 min read
  • rag
  • query
  • search

Follow-up: “what about the other plan?” Your index has never heard of “the other plan.” The previous turn said “Pro versus Basic,” but retrieve only sees the latest string, so it searches a pronoun.

Query rewriting is a pre-search transform: take the user text (and maybe the thread), emit one or more search strings that the index can actually match. Then retrieve as usual.

Key takeaways

  • Contextualize follow-ups with history before you embed or BM25.
  • Multi-query covers mixed intents. Fuse the hit lists. Cap how many.
  • HyDE embeds a hypothetical answer. Optional, extra latency, can retrieve fluent fiction.
  • Do not rewrite SKUs, error codes, or emails into prose.
  • Rewriting is another LLM call on the critical path. Time it out and fall back to the raw query.

The index is literal. Users are not.

RAG retrieves from a query. Chat users do not speak in queries. They say “that one,” “the error from before,” “same but for EU.” Your conversation state has the missing nouns. The BM25 postings list does not.

This is the same problem as search-as-you-type in a product UI, with a worse default: if you embed the raw follow-up, cosine will wander into unrelated “plan” documents (meal plans, capacity plans, incident plans).

LangChain’s retrieval docs cover the same control flow (rewrite, then retrieve) if you want a library instead of 40 lines: retrieval. You do not need the library. You need the control flow.

Query rewriting styles

Style Input → output Use when Tax
Contextualize Thread + last user → one standalone question Chat follow-ups, pronouns One short completion
Multi-query One question → 3–5 search strings Mixed or underspecified asks Several retrieves, then fusion
HyDE Question → hypothetical passage, embed that Short questions, paraphrase-heavy corpora Generator can invent facts the corpus never had
Split identifiers “refund for ORD-18422” → filter orderId + query “refund” SKUs, tickets, emails You must detect the id, not poetically expand it

Start with contextualize. Add multi-query when eval shows mixed intents. Treat HyDE as an experiment.

Contextualize with history

Keep the rewrite boring. The model is a query compiler, not an answerer.

type Turn = { role: 'user' | 'assistant'; content: string };

const REWRITE_SYSTEM = `Rewrite the last user message as a standalone search query for a docs index.
- Resolve pronouns using the conversation.
- Keep identifiers (SKU, order id, error code) verbatim.
- Do not answer the question.
- Output one line of search text, nothing else.`;

async function standaloneQuery(history: Turn[], lastUser: string): Promise<string> {
  if (history.length === 0) return lastUser;

  const completion = await generate({
    temperature: 0,
    maxTokens: 80,
    messages: [
      { role: 'system', content: REWRITE_SYSTEM },
      ...history.slice(-6),
      { role: 'user', content: lastUser },
    ],
  });

  const q = completion.text.trim().split('\n')[0] ?? lastUser;
  return q.length === 0 ? lastUser : q;
}

Failure: the rewriter “helpfully” turns ORD-18422 into “the customer’s previous order about billing.” BM25 loses the only term that mattered. Hybrid search cannot save a query that deleted the SKU.

Pass the last few turns, not the whole novel. Rewriting does not need the assistant’s essay, only enough nouns. Conversation state in chat UIs is where you store that thread.

Anthropic’s prompting best practices apply: short instructions, explicit negatives (“do not answer”), tight max tokens.

Multi-query and HyDE

Multi-query: ask for three phrasings (“refund window,” “return period,” “money-back days”), retrieve each with a small k, fuse with RRF, then rerank. Good for FAQ corpora where authors used one synonym and users used another. Bad when you generate 12 queries and multiply tail latency by 12.

HyDE (Hypothetical Document Embeddings, Gao et al. 2022): generate a fake answer paragraph, embed the paragraph, search for real chunks near that embedding. The bet is that document-document similarity beats query-document similarity for short questions.

HyDE’s failure mode is specific: the hypothetical passage states a confident policy your company does not have, and retrieval then finds the closest real cousin, which may still be wrong. Never show the hypothetical text to the user. Never pack it. It is a search probe, not a source.

async function hydeProbe(question: string): Promise<string> {
  const completion = await generate({
    temperature: 0.3,
    maxTokens: 180,
    messages: [
      {
        role: 'system',
        content:
          'Write a short hypothetical docs paragraph that would answer the question. It may be wrong. Do not cite sources.',
      },
      { role: 'user', content: question },
    ],
  });
  return completion.text;
}

Run HyDE only if contextualize plus hybrid still misses on the golden set. It is not a default.

Latency, fallback, and what not to rewrite

Rewriting adds a completion before retrieve before generate. Users feel that as dead air unless you stream a thinking state, and even then you should:

  1. Timeout the rewrite (a few hundred milliseconds of budget, or a tight maxTokens).
  2. Fall back to the raw last user string on timeout or empty output.
  3. Skip rewrite when the message looks like an identifier (/^[A-Z]{2,}-?\d{3,}$/ and friends) or when there is no history.
  4. Abort on new input. AbortController.

Do not rewrite:

  • SKUs, order ids, trace ids, emails, hashes
  • Quoted error strings from logs (BM25 wants the tokens)
  • Questions that are already standalone

Eval rewrite as its own layer: compare retrieval hit rate with raw vs rewritten queries on a labeled chat-follow-up set. If rewrite helps the assistant sound smarter but drops hit rate, it is hurting the product.

Interview answer (out loud)

“Query rewriting turns chat-shaped text into search-shaped text. I first contextualize follow-ups with recent history so pronouns resolve, while copying identifiers verbatim. If eval shows mixed intents I add a small multi-query and fuse with RRF. HyDE is optional: embed a hypothetical paragraph, never show it. The rewrite call is budgeted and falls back to the raw query, because a slow rewriter in front of RAG is just extra latency.”

Further reading (primary sources)

Related guides