Frontend Beauty
ESC

Type to search the knowledge base.

Hybrid Search: BM25 and Vectors

Keyword search still wins on SKUs, error codes, and rare names. Hybrid retrieval blends BM25 with vectors so neither failure mode owns you.

intermediate7 min read
  • search
  • bm25
  • rag

ORD-18422 is a terrible embedding query and a perfect keyword query. Dense vectors smear rare tokens into a neighborhood of unrelated orders. BM25 treats the string as a term that almost no document contains, which is exactly what you want.

Hybrid search runs both, then merges the ranked lists. Neither retriever is allowed to own the failure mode.

Key takeaways

  • BM25 wins on identifiers, error codes, SKUs, and rare proper names.
  • Vectors win on paraphrase (“how do I get my money back” vs “refunds”).
  • Reciprocal rank fusion merges lists by rank, so you do not have to calibrate score scales.
  • Skip vectors when the query is an id you can exact-match. Skip BM25 when the corpus is paraphrase-only.
  • Postgres tsvector is a lexical ranker (ts_rank), not BM25. Measure with labeled queries.

Two different bugs

Vector similarity search fails closed on terms the embedding model barely saw. Keyword search fails closed on questions that share no tokens with the document.

Support corpora have both: “what’s your return window?” and “ticket INC-90311.” If you only embed, agents paste the ticket id and retrieve a random incident that “feels” similar. If you only BM25, the paraphrase question misses the policy titled “Returns and refunds.”

Elastic’s hybrid search explainer and OpenSearch’s hybrid search docs are the same idea in two engines: lexical plus neural, then a combine step.

What BM25 is scoring

BM25 (Best Matching 25) is a bag-of-words ranker. For each query term it rewards term frequency in the document, damps that reward as frequency grows, penalizes long documents, and boosts terms that are rare in the corpus (IDF).

It does not know that “return” and “refund” are cousins. It knows that ORD-18422 appears in three documents and that is a strong clue.

Where that ranker actually lives:

Engine Lexical ranker
Elasticsearch, OpenSearch, Typesense BM25 (or a BM25 variant)
Postgres built-in FTS tsvector + ts_rank / ts_rank_cd. Not BM25.
Postgres + an extension (ParadeDB pg_search, and similar) BM25 inside Postgres

If you are on stock Postgres, say “keyword score” or “ts_rank,” not “BM25.” The sketch below is a schematic BM25-like keyword score: same knobs (saturated TF, IDF, length penalty), not a Lucene port.

/** Schematic BM25-like keyword score. Not Lucene's BM25. */
function keywordScore(
  queryTerms: string[],
  docTerms: string[],
  avgDocLen: number,
  df: Map<string, number>,
  nDocs: number,
  k1 = 1.2,
  b = 0.75,
): number {
  const tf = new Map<string, number>();
  for (const t of docTerms) tf.set(t, (tf.get(t) ?? 0) + 1);
  const docLen = docTerms.length || 1;
  let score = 0;
  for (const term of queryTerms) {
    const f = tf.get(term) ?? 0;
    if (f === 0) continue;
    const n = df.get(term) ?? 0;
    const idf = Math.log(1 + (nDocs - n + 0.5) / (n + 0.5));
    const denom = f + k1 * (1 - b + b * (docLen / avgDocLen));
    score += idf * ((f * (k1 + 1)) / denom);
  }
  return score;
}

function tokenize(s: string): string[] {
  return s.toLowerCase().split(/[^a-z0-9+-]+/).filter(Boolean);
}

function topKKeyword(
  query: string,
  docs: { id: string; text: string }[],
  k: number,
): string[] {
  const q = tokenize(query);
  const tokenized = docs.map((d) => ({ id: d.id, terms: tokenize(d.text) }));
  const nDocs = tokenized.length || 1;
  const avgDocLen =
    tokenized.reduce((s, d) => s + d.terms.length, 0) / nDocs;
  const df = new Map<string, number>();
  for (const d of tokenized) {
    for (const t of new Set(d.terms)) df.set(t, (df.get(t) ?? 0) + 1);
  }
  return tokenized
    .map((d) => ({
      id: d.id,
      score: keywordScore(q, d.terms, avgDocLen, df, nDocs),
    }))
    .sort((a, b) => b.score - a.score)
    .slice(0, k)
    .map((r) => r.id);
}

Production should call Elasticsearch, OpenSearch, or a real BM25 library. The function is here so the merge has two honest lists, not a comment that says “magic BM25.”

Hybrid search as two lists, then a merge

  1. Normalize the user text (trim, maybe keep punctuation inside ids).
  2. Run keyword top-k on the same chunk corpus the vectors use. Same ids.
  3. Embed the query, run ANN top-k.
  4. Merge. The simple merge is reciprocal rank fusion.
  5. Hydrate texts from ids. Optionally rerank the fused list. Reranking for RAG.

Same corpus is not optional. Fusing “page-level BM25” with “paragraph-level vectors” makes ranks incomparable.

Reciprocal rank fusion

Raw BM25 scores and cosine scores live on different scales. Do not add them until you have a reason and an eval. RRF ignores scores and uses ranks.

Cormack, Clarke, and Büttcher (SIGIR 2009) scored a document as the sum of 1 / (k + rank) across lists, with k = 60 chosen in a pilot and left alone. k is a smoothing constant, not magic. Smaller k favors top ranks more aggressively.

type Hit = { id: string; rank: number }; // rank is 1-based
type Chunk = { id: string; text: string };

function rrf(lists: Hit[][], k = 60): { id: string; score: number }[] {
  const scores = new Map<string, number>();
  for (const list of lists) {
    for (const { id, rank } of list) {
      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + rank));
    }
  }
  return [...scores.entries()]
    .map(([id, score]) => ({ id, score }))
    .sort((a, b) => b.score - a.score);
}

function toRanked(ids: string[]): Hit[] {
  return ids.map((id, i) => ({ id, rank: i + 1 }));
}

function hybridRetrieve(
  query: string,
  corpus: Chunk[],
  annTopK: (q: string, k: number) => string[],
  k = 8,
): Chunk[] {
  const keywordIds = topKKeyword(query, corpus, k);
  const vectorIds = annTopK(query, k);
  const fused = rrf([toRanked(keywordIds), toRanked(vectorIds)]);
  const byId = new Map(corpus.map((c) => [c.id, c]));
  const out: Chunk[] = [];
  for (const row of fused) {
    const chunk = byId.get(row.id);
    if (chunk) out.push(chunk);
    if (out.length >= k) break;
  }
  return out;
}

// Keyword list found sku-18422 at rank 1. Vector list buried it.
// After RRF, hydrate texts from the fused ids, not from one side alone.

A document that ranks well in both lists rises. A document that ranks well in one list still survives. That is the point of hybrid: each retriever can rescue the other.

Weighted score fusion (normalize then α * keyword + (1-α) * cosine) can beat RRF after you tune α on labeled data. Start with RRF. It needs no α.

When to skip a side

Query shape Lean on
Order id, SKU, error code, email, hash Keyword or exact filter. Vectors optional.
Natural-language how-to, paraphrase Vectors, then keyword as backup.
Mixed: “refund for ORD-18422” Both. Consider splitting the id into a metadata filter.
Empty index, brand-new corpus Keywords until you have embeddings and a sanity check.

Do not rewrite a SKU into a sentence before the keyword ranker. Query rewriting that turns ORD-18422 into “customer order details” is how you throw away the only useful token. Keep identifiers intact; expand the prose part.

Eval the two retrievers separately, then the fusion. Hit rate and MRR on a labeled set will tell you if keywords were already enough. RAG evaluation metrics is that split. This page will not invent an NDCG lift.

Frontend-wise, run the two searches in parallel and bound the wait. The merge is cheap. The slow part is the embedding call and the ANN round trip, which you should abort if the user types again: AbortController.

Interview answer (out loud)

“Hybrid search runs a keyword ranker and vector retrieval on the same chunk ids, then merges. BM25 (in ES or OpenSearch; not stock Postgres ts_rank) catches SKUs, error codes, and rare names. Vectors catch paraphrase. I merge with reciprocal rank fusion, 1 / (k + rank) with k around 60, so I do not have to calibrate score scales. I would skip vectors for pure identifier lookup, and I would measure retrieval hit rate on labeled queries rather than quoting a made-up NDCG number.”

Further reading (primary sources)

Related guides