Frontend Beauty
ESC

Type to search the knowledge base.

Reranking for RAG

Retrieve broadly, then rerank with a cross-encoder or a small model so the prompt gets the 5 chunks that actually answer the question.

intermediate6 min read
  • rag
  • rerank
  • search

Top-20 from ANN is a candidate list. The prompt only has room for four. Reranking is the editor: a second model that looks at the question and each candidate together and reorders them.

If you pack the first four cosine hits, you are letting a cheap bi-encoder choose the context window. That model never compared the question to the paragraph in one pass.

Key takeaways

  • Retrieve wide (keyword + vectors), rerank, then cut to pack-n. Sort before any score floor.
  • Bi-encoders embed query and document separately. Cross-encoders score the pair.
  • Rerank costs latency and money on every question. Cap candidate count.
  • A reranker cannot invent a chunk you failed to retrieve.
  • Score thresholds are model-specific. Prefer a fixed top-n plus a refusal path.

Two encoders, two jobs

A bi-encoder (your embedding model) encodes the query once and each document once. Similarity is a dot product. That is why ANN is fast: documents were encoded at index time.

A cross-encoder (typical reranker) takes (query, document) as one input and emits a relevance score. It sees word alignment, negation, and “except when” clauses that cosine smears. It is too slow to run on the whole corpus, which is why it only sees the candidate list.

Method Input Strength Cost
Bi-encoder query vec · doc vec Cheap ANN over the corpus Smears negation and “except”
Cross-encoder (query, doc) pair Word alignment; the editor O(candidates), extra latency
LLM-as-rerank Prompt with passages No extra model if you already generate Slower, less stable scores, judge drift

Cohere’s Rerank API is this second stage as a service: query plus documents in, ordered scores out. You can also run an open cross-encoder such as BAAI/bge-reranker-v2-m3 next to your API, or call Voyage’s reranker. The architecture does not change.

LLM-as-rerank (prompt: “score these passages”) works in a pinch. It is usually slower and less stable than a dedicated rerank model: the scores drift when you change the judge, and you pay generate prices for a sort. Use it for experiments, pin the judge, and do not confuse it with evals.

Retrieve broadly, then edit

  1. Build a candidate set. A common starting heuristic is 20-50 chunks (start at 20, raise if labeled recall@20 is still low): BM25 ∪ vectors, already tenant-filtered. Hybrid search.
  2. Deduplicate near-copies (same url + high text overlap).
  3. Send (query, candidate texts) to the reranker. Ask for the top 3-8.
  4. Pack those into the prompt under a tight token budget. Context engineering.
  5. If the best score is garbage, refuse. Do not pack “something” because the UI hates silence.

Rerank first, then cut. Do not break on a relevance floor unless the list is already sorted descending. Vendor responses are usually sorted. Do not assume they are.

type Candidate = { id: string; text: string; retrieveScore: number };

type RerankHit = { index: number; relevance_score: number };

type RerankRequest = {
  model: string;
  query: string;
  documents: string[];
  top_n: number;
};

type RerankResponse = {
  results: RerankHit[];
};

async function retrieveThenRerank(opts: {
  query: string;
  retrieve: (q: string, k: number) => Promise<Candidate[]>;
  rerank: (body: RerankRequest) => Promise<RerankResponse>;
  retrieveK: number;
  packN: number;
  minRelevance?: number; // optional floor, applied after sort
}): Promise<Candidate[]> {
  const candidates = await opts.retrieve(opts.query, opts.retrieveK);
  if (candidates.length === 0) return [];

  const response = await opts.rerank({
    model: 'rerank-v5.0-pro', // pin the vendor model id in config
    query: opts.query,
    documents: candidates.map((c) => c.text),
    top_n: candidates.length, // get scores for the whole list; we sort and cut
  });

  const ranked = [...response.results]
    .filter(
      (h) =>
        Number.isInteger(h.index) &&
        h.index >= 0 &&
        h.index < candidates.length,
    )
    .sort((a, b) => b.relevance_score - a.relevance_score);

  const floor = opts.minRelevance;
  const usable =
    floor == null
      ? ranked
      : ranked.filter((h) => h.relevance_score >= floor);

  return usable.slice(0, opts.packN).map((h) => candidates[h.index]);
}

A schematic Cohere-shaped payload looks like this. Field names drift; treat it as the contract (query + documents in, { index, relevance_score } out), not a pinned SDK:

{
  "model": "rerank-v5.0-pro",
  "query": "How long is the refund window for billing errors?",
  "documents": [
    "Refunds: digital goods, 14 days.",
    "Billing errors: full refund within 30 days.",
    "Shipping: 5-7 business days."
  ],
  "top_n": 3
}
{
  "results": [
    { "index": 1, "relevance_score": 0.91 },
    { "index": 0, "relevance_score": 0.44 },
    { "index": 2, "relevance_score": 0.11 }
  ]
}

minRelevance is dangerous if you copy it from another model card. Calibrate on your labeled set, or skip the threshold and always take top-n, refusing only when retrieve returned nothing. If you do use a floor, apply it to the sorted list. Breaking on the first low score in vendor order is how you drop a high-scoring hit that arrived late.

The latency and cost tax

Rerank sits on the user-critical path, after retrieve and before generate. You pay:

  • Another model call on every question, proportional to candidate count (and document length; long docs get truncated or chunked inside the reranker).
  • Tail latency. p95 is retrieve p95 plus rerank p95 plus generate TTFT. Streaming the answer does not hide the rerank round trip before the first token.
  • Truncation. If you pass a 10k-character candidate, the reranker may only see the start. That is another reason chunking still matters.

Mitigations: keep retrieveK modest (20 is a starting heuristic, not a law), run hybrid retrieve in parallel, skip rerank for identifier queries that already exact-matched one row, cache rerank results for identical (query, candidate ids) pairs.

Abort in-flight rerank when the user sends a new message. Same discipline as any other fetch: AbortController.

When reranking cannot save you

Rerank reorders the list it was given. It cannot:

  • Fetch the exception paragraph you never indexed
  • Undo a missing tenant filter (and must not be used as one)
  • Fix a query that still says “the other plan” with no history
  • Make four packed chunks span a 40-page argument

If labeled retrieval hit rate is poor before rerank, fix ingest, chunking, hybrid, or query rewriting. Rerank is an editor, not a librarian.

Watch for score addiction. Teams plot relevance histograms, pick a cutoff that “looks clean,” and then a model upgrade shifts the distribution. Keep the cutoff in config, version it, and re-calibrate on the golden set when the reranker model id changes.

Interview answer (out loud)

“I retrieve a wide candidate list with hybrid search, then rerank with a cross-encoder so the prompt only gets the few chunks that actually match the question. Bi-encoders are fast because they encode documents offline. Cross-encoders are accurate because they score the pair. I sort by relevance, then take top-n; I do not threshold an unsorted list. Rerank adds latency, so I cap candidates. If the right paragraph was never retrieved, rerank cannot help, and I would not use it as a security filter.”

Further reading (primary sources)

Related guides