Frontend Beauty
ESC

Type to search the knowledge base.

Embeddings and Similarity Search

Nearest-neighbor search over embeddings is fast guesswork. Distance metrics, top-k, and why the nearest chunk can still be the wrong chunk.

intermediate6 min read
  • embeddings
  • search
  • rag

Top-1 cosine is not “the answer.” It is “the least dissimilar needle we stored.” If the needle is a sibling policy, or last year’s PDF, the generator will still treat it as evidence.

Vector similarity search is nearest-neighbor lookup in embedding space. You embed the query with the same model you used at index time, you ask for the k closest stored vectors, you hope closeness means relevance. Hope is not a contract.

Key takeaways

  • Exact k-NN scores every stored vector. ANN skips most of them on purpose.
  • top-k is a candidate budget, not a confidence interval.
  • The nearest chunk can be the wrong clause, the wrong year, or an identifier the embedder smeared.
  • Store unit vectors at index time. Do not re-normalize the corpus on every query.
  • Mixing embedding models in one index silently corrupts ranking.

Geometry lives on the sibling page

Embeddings explained is cosine, dot product, L2, and why nearby is not “understood.” This page is the lookup: exact versus approximate search, how k is chosen, and why the winner can still be the wrong paragraph.

Use the distance the embedding model was trained with. OpenAI’s embeddings guide and Gemini’s embeddings API both assume you compare vectors from one model. Do not mix text-embedding-3-large rows with a MiniLM index.

Metric What the index ranks Practical note
Cosine Angle (direction) Default for many text APIs
Dot product Direction and magnitude Same ranking as cosine if both sides are L2-normalized
L2 (Euclidean) Straight-line distance Common for images and some older text models

The index is built for one metric. Changing metric means a rebuild.

Similarity search: exact k-NN is a scan, ANN is a skip

Exact k-NN: score the query against every stored vector, sort, take k. Correct for the metric you chose. Fine for thousands of chunks in memory. Death for millions on every keystroke.

ANN (approximate nearest neighbor): a graph or inverted file that usually finds the true neighbors. You trade a miss rate for latency.

HNSW (hierarchical navigable small world) is the common graph. Layer 0 holds every point with short edges. Higher layers are sparse, with longer edges. Search starts at an entry point on the top layer, walks greedily to a local minimum, and drops one layer. On layer 0 it expands a candidate list of size efSearch and returns the closest k. Raising efSearch explores more neighbors before stopping: more recall, more latency. Greedy walks can miss a better basin. That miss is the approximation.

IVF (inverted file) clusters vectors (k-means) into lists and only scores the nprobe lists whose centroids are closest to the query. Raising nprobe searches more lists. IVF needs data to train the centroids; building it on an empty table is a no-op. After many inserts, clusters drift and recall falls until you rebuild.

If your corpus is a few thousand doc chunks, skip ANN. Brute force is honest and fast enough. A vector database is this algorithm plus persistence, filters, and ops. Do not wrap a hosted index until brute force on a sample of your queries shows that embeddings help at all.

Top-k is a candidate list

k is how many neighbors you are willing to inspect, not a probability that any of them answer the question.

  • k = 1 is a demo. The nearest title often beats the deciding clause.
  • k too small misses the exception paragraph that sat at rank 7.
  • k too large fills the prompt with cousins. Packing then becomes the bottleneck.

Always retrieve more than you pack, then filter, fuse, or rerank. Hybrid search and reranking exist because cosine is a blunt instrument.

Score thresholds (“only use hits above 0.8”) look scientific and fail across models and corpora. A 0.8 on one embedding space is not a 0.8 on another. Prefer a fixed k plus a rerank, and refuse when the editor scores everything as irrelevant.

A teaching scan on stored unit vectors

Normalize once at index time. The query is normalized once at the API boundary. Scoring is a dot product. Re-L2-normalizing every document on every keystroke is a teaching bug, not an index.

type Item = { id: string; text: string; unit: number[] }; // L2-normalized at index time

function dot(a: number[], b: number[]): number {
  if (a.length !== b.length) throw new Error('dim mismatch');
  let s = 0;
  for (let i = 0; i < a.length; i++) s += a[i] * b[i];
  return s;
}

function exactTopK(
  queryUnit: number[],
  items: Item[],
  k: number,
): { item: Item; score: number }[] {
  // Teaching scan, not an index: O(n) over the whole corpus.
  if (k <= 0) return [];
  return items
    .map((item) => ({ item, score: dot(queryUnit, item.unit) }))
    .sort((a, b) => b.score - a.score)
    .slice(0, k);
}

// Failure: top-1 is a near-duplicate title, not the clause that answers the question.
// Retrieve k=8, then rerank or fuse with keywords.

Asymmetric models (query encoder ≠ document encoder) still compare in one space, but you must use the query tower at ask time and the document tower at index time. Using the document tower for the question is a silent quality drop.

Identical questions should not re-embed on every keystroke. Cache by model id plus text; the HTTP half is frontend caching strategies.

Why the nearest chunk is still wrong

  1. The right answer was never indexed. ANN cannot retrieve a paragraph you did not store.
  2. ANN missed it. The true neighbor was in a basin the greedy walk never entered. Exact k-NN on the same vectors would have returned it. That is the recall you plot against efSearch / nprobe.
  3. The query is an identifier. ORD-18422 is a keyword problem. Embeddings smear rare tokens.
  4. Same topic, different decision. “Admin SSO setup” and “end-user password login” sit close. Cosine loves topical overlap, not the role the user is in.
  5. Stale duplicate. Last year’s rate-limit page embeds like this year’s. Metadata dates are not optional.
  6. k is too small, or too large. k=1 misses. k=50 fills the prompt with cousins.

Nearest is a ranking, not a certificate. Treat it like a search result page, not a database SELECT.

Interview answer (out loud)

“Vector similarity search embeds the query with the same model as the index, then returns the k nearest chunks. Exact k-NN scans everything. ANN (HNSW, IVF) skips most vectors and can miss the true neighbor; I tune that miss with efSearch or nprobe. top-k is a candidate list, not confidence. Nearest is not correct: identifiers need keywords, topical cousins outrank the deciding clause, and ANN can drop the right row. I retrieve a wider k, then hybrid-fuse or rerank, and I never treat top-1 as the answer.”

Further reading (primary sources)

Related guides