Frontend Beauty
ESC

Type to search the knowledge base.

Embeddings Explained

An embedding is a vector that places text near similar text. How cosine similarity works in practice, and why it is not understanding.

beginner6 min read
  • embeddings
  • vectors
  • rag

“Refund policy” and “how do I get my money back” can sit close in vector space and still be different intents. One wants a document. The other wants a workflow, an order id, and a permission check. Text embeddings measure neighborhood, not understanding.

An embedding model maps a string to a list of numbers. Similar strings (paraphrases, shared jargon) tend to land nearby. That is enough to retrieve candidate chunks. It is not enough to know what the user is allowed to do, whether the chunk is current, or whether the question is about this product’s refunds.

If you treat cosine similarity as “the model got the meaning,” you will ship a search that is fluent and wrong.

Key takeaways

  • An embedding is a vector. Retrieval is nearest-neighbor search in that space, usually by cosine similarity.
  • Nearby is “looks like,” not “answers.” Rank, then read, then generate.
  • Vectors from model A are not comparable to vectors from model B. Re-embed if you change models.
  • Some stacks use different encoders for queries and documents. Do not mix those spaces either.

Text embeddings: a vector and a cosine

Picture each string as a point on a high-dimensional sphere (hundreds or thousands of axes). You do not inspect the axes. You compare angles.

Cosine similarity is the cosine of the angle between two vectors. 1 means they point the same way. 0 means orthogonal. Negative values show up if the space allows them, which some embedding models do.

If you L2-normalize the vectors (length 1), cosine equals the dot product. Many indexes store normalized vectors and use inner product for that reason.

function cosineSimilarity(a: readonly number[], b: readonly number[]): number {
  if (a.length !== b.length) {
    throw new Error(`embedding length mismatch: ${a.length} vs ${b.length}`);
  }

  let dot = 0;
  let normA = 0;
  let normB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }

  const denom = Math.sqrt(normA) * Math.sqrt(normB);
  if (denom === 0) return 0;
  return dot / denom;
}

function topK(
  query: number[],
  docs: { id: string; vector: number[] }[],
  k: number,
): { id: string; score: number }[] {
  return docs
    .map((d) => ({ id: d.id, score: cosineSimilarity(query, d.vector) }))
    .sort((x, y) => y.score - x.score)
    .slice(0, k);
}

// Failure mode: comparing a 1536-d OpenAI vector to a Gemini vector.
// Length mismatch should throw. Silent padding would be worse.

This is the same geometry as “nearest neighbor in a list,” not a neural network in your app. Production search uses an ANN index (vector databases) so you do not scan every doc. The score still means “close in this embedding space.”

Other distances exist (L2, dot product without cosine). They agree when vectors are normalized and centered in particular ways. They disagree when they are not. Pick the metric the model was trained for, and do not mix metrics in one index.

Same model, same space

An embedding is not “the meaning of the sentence” in the abstract. It is the meaning according to this checkpoint, this dimensionality, this instruction prefix.

Rules that save incidents:

  1. One embedding model per index. Changing models means re-embedding the corpus. Mixing leftover rows from the old model poisons neighbors.
  2. Same dimensionality. A 768-d vector and a 1536-d vector are different objects.
  3. Same preprocessing. Lowercasing, stripping markdown, chopping at 8k tokens: do it on query and document, or you invented a mismatch.
  4. Record the model id on every stored vector. Future you will thank you.
type EmbeddedChunk = {
  id: string;
  text: string;
  vector: number[];
  embeddingModel: string;
};

async function embedQuery(
  text: string,
  model: string,
): Promise<number[]> {
  const res = await fetch('/api/embeddings', {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ model, input: text }),
  });
  if (!res.ok) throw new Error(`embed ${res.status}`);
  const body = (await res.json()) as { vector: number[]; model: string };
  if (body.model !== model) {
    throw new Error('embedding model mismatch');
  }
  return body.vector;
}

Provider docs (OpenAI, Gemini) show the request as input plus a model name. The response is a float array. There is no “understanding” field.

Query vectors vs document vectors

A 12-word question and a 400-word policy section are different shapes. Asymmetric retrieval models train two encoders: one for queries, one for documents, so “how do I get my money back” can still land near the refund section.

Many API embedding models use a single encoder for both. They still behave better if you:

  • Embed documents as the chunk you will actually show (heading path included).
  • Embed queries as the user’s language, not as a fake document.
  • Optionally prefix with an instruction the vendor documents (query: vs passage:), and keep those prefixes consistent.

Do not embed the query with query: and the docs with nothing if the model card says both sides need a prefix. That is a silent recall drop.

If the user says “the other plan,” the vector for those three words will not match a chunk titled “Pro vs Team.” Query rewriting is a separate step. Embeddings will not invent the missing noun.

Nearby is not the answer

Cosine 0.82 against a chunk titled “Refund policy” does not mean:

  • The chunk answers this order
  • The policy is in force
  • The user may see that tenant’s document
  • The number in the chunk is still correct

It means: among the vectors you stored, this one was least unlike the query vector. RAG is retrieve, pack, generate, cite. The embedding is only the retrieve hint.

Concrete mismatches:

Query Nearby chunk Why it is still wrong
how do I get my money back Refund policy (retail, 2019) Wrong product line, stale
error E_CHARGE_9 essay about “charges” Jargon lost; keyword search would win
cancel my Pro plan “How to cancel a reservation” Same verb, different domain
invoice INV-18422 some other invoice-shaped text Identifiers are poor embedding queries

For SKUs, error codes, and ids, hybrid search (keyword plus vectors) beats cosine alone. That is the next page in this cluster: embeddings and similarity search.

Frontend analogy: this is autocomplete ranking, not comprehension. A typeahead can put “Apple” next to “Apple Pay” because the strings are close. It does not know you meant the fruit.

What you should not do

Store embeddings of secrets. A vector can leak enough to be awkward, and the source text in metadata definitely can. Treat the index like a document store with ACLs.

Use chat-model “embeddings” you invented by averaging hidden states from a completion API. Use an embedding endpoint.

Threshold superstition. “Drop scores below 0.75” is not portable across models. Calibrate on your corpus or rerank.

One vector per handbook. A 40-page PDF as a single point retrieves the whole book or none of it. Chunk first.

Out loud, in an interview

“A text embedding is a vector from a specific model. I retrieve by cosine similarity, which is neighborhood in that space, not understanding. I only compare vectors from the same model and the same preprocessing. Queries and documents may need different prefixes. A high score is a candidate chunk, not an answer, and identifiers still want keyword search. I would never mix leftover vectors after a model change.”

Further reading (primary sources)

Related guides