Frontend Beauty
ESC

Type to search the knowledge base.

Semantic Cache for LLM Apps

A semantic cache returns a previous answer when a new query is close enough. Great for FAQs. Dangerous for personalized or time-sensitive asks.

intermediate6 min read
  • caching
  • embeddings
  • cost

“What’s the weather?” matching yesterday’s answer is not a clever semantic cache. It is a cache with no TTL. Near-duplicate FAQ questions are the happy path: “reset my password” and “how do I change my password” can share a vetted reply. “What’s my order status?” cannot.

A semantic cache embeds the query, looks for a close neighbor, and returns the stored completion if the score clears a threshold. It sits in front of the model, not inside it. Exact-match first, then similarity, then live generation. Never cache a tool write. Never share entries across tenants.

Key takeaways

  • Semantic cache: embed the query, nearest neighbor, threshold, return a previous answer.
  • Exact-match (normalized string) is cheaper and safer. Try it first.
  • Key by tenant, locale, prompt version, and cacheability. TTL is mandatory for anything that can go stale.
  • Personalized, time-sensitive, and side-effecting requests are cache misses on purpose.
  • A threshold that is too low is a wrong-answer machine. Log hit source and sample the hits.

How a semantic cache decides to skip the model

Three lookups, in order:

  1. Exact. Normalize whitespace and casing, hash, Redis get. FAQs with copy-paste questions die here.
  2. Semantic. Embed with the same model you indexed. Query a small vector set of answers you are willing to replay.
  3. Live. Call the LLM, maybe store the result if a policy says the intent is cacheable.

This is not prompt caching. Prompt cache reuses a prefix on the provider (KV). Semantic cache reuses an answer you already generated. Embeddings here are a lookup key, not “understanding.”

type CacheQuery = {
  tenantId: string;
  userId: string;
  locale: string;
  promptVersion: string;
  text: string;
  intent: 'faq' | 'account' | 'weather' | 'other';
};

type Hit = {
  text: string;
  source: 'exact' | 'semantic' | 'live';
};

const SEMANTIC_MIN = 0.95;

function exactKey(q: CacheQuery) {
  const norm = q.text.trim().toLowerCase().replace(/\s+/g, ' ');
  return ['llm', q.tenantId, q.locale, q.promptVersion, hash(norm)].join(':');
}

export async function answerWithCache(
  q: CacheQuery,
  deps: {
    redis: { get(k: string): Promise<string | null>; set(k: string, v: string, ttlSec: number): Promise<void> };
    embed: (text: string) => Promise<number[]>;
    search: (v: number[], filter: object) => Promise<Array<{ score: number; answer: string; userId?: string; expiresAt: number }>>;
    live: (q: CacheQuery) => Promise<string>;
  },
): Promise<Hit> {
  if (q.intent === 'account' || q.intent === 'weather') {
    return { text: await deps.live(q), source: 'live' };
  }

  const cached = await deps.redis.get(exactKey(q));
  if (cached) return { text: cached, source: 'exact' };

  const vector = await deps.embed(q.text);
  const neighbors = await deps.search(vector, {
    tenantId: q.tenantId,
    locale: q.locale,
    promptVersion: q.promptVersion,
    cacheable: true,
  });

  const best = neighbors[0];
  if (
    best &&
    best.score >= SEMANTIC_MIN &&
    best.expiresAt > Date.now() &&
    (!best.userId || best.userId === q.userId)
  ) {
    return { text: best.answer, source: 'semantic' };
  }

  const text = await deps.live(q);
  if (q.intent === 'faq') {
    await deps.redis.set(exactKey(q), text, 60 * 60 * 24);
    // Insert embedding + answer into the tenant namespace with the same TTL.
  }
  return { text, source: 'live' };
}

0.95 is not a law. It is a starting knob you tune on a labeled set of “same intent / different intent” pairs. Publish it like you would a ranking cutoff. If you cannot find a threshold that separates those pairs, you do not have a cache, you have a coin flip.

Keys, tenants, and TTL

A cache entry is a product decision wearing a vector.

Dimension Why it is in the key / filter
Tenant id Cosine does not know GDPR.
Locale “Reset password” in DE is not the EN article.
Prompt version Yesterday’s answer may violate today’s policy.
User id (optional) Only for entries that are allowed to be personal, which is rare.
TTL Weather, prices, incidents, anything with a clock.
cacheable: false Default for account, payments, medical, legal-advice-shaped intents.

Namespace vectors per tenant. A shared global FAQ index is how one customer’s custom contract wording shows up in another’s “what is included?” hit.

TTL is not optional because embeddings are “close.” Close to yesterday is still yesterday. CDN mental model: stale content is a correctness bug (what is caching). For FAQs, a day is often fine. For “is there an outage?”, minutes, or do not cache.

When you change the system prompt, bump promptVersion and let old entries die. Do not try to “invalidate by meaning.”

When similarity lies

Paraphrase vs opposite. “Can I refund after 30 days?” and “Can I still refund after 30 days if I opened the box?” can sit near each other and need different answers. High threshold, or require shared entities (product SKU) in metadata before you hit.

Negation. “disable 2FA” vs “do not disable 2FA” may be closer than you want. Keyword overlap plus embeddings (hybrid) is safer than vectors alone.

Personal slots. “What is my tracking number?” will neighbor someone else’s tracking question. If the cached answer contains a number, it is a leak. Policy: if the live answer matches /\d{5,}/ or an email, mark cacheable: false.

Tool writes. Caching “sure, I cancelled that” is how you skip the cancel. Semantic cache is for pure generation. Side effects go through tools with idempotency, never through Redis.

Prompt injection in a cached answer. If a retrieved page poisoned a live call and you stored the result, you now serve the poison cheaply. Only cache intents that did not include untrusted retrieved HTML, or cache after a sanitizer / allowlisted corpus.

Threshold theater. Lowering the score to raise hit rate is a quiet quality regression. Track source=semantic in traces and sample them into the eval set.

This is a cost lever after you already did prompt cache, smaller models, and “don’t dump the PDF.” It is not the first lever.

Footguns

  • One global cosine cutoff for all intents. FAQs and troubleshooting need different numbers.
  • Caching errors. A 500 page stored as the answer will replay until TTL. Only store successful, schema-valid completions.
  • Forgetting to filter promptVersion. You shipped a safer refusal, the cache still jokes.
  • Storing the user question with PII in the vector metadata. The cache is another production database. Redact.
  • Using a different embedding model at read time. Neighbor scores become noise. Pin the model id.

Interview answer (out loud)

“A semantic cache embeds the new query and returns a previous answer if it is close enough, after an exact-match lookup. I key by tenant, locale, and prompt version, set a TTL, and I only store FAQ-like intents. Account, weather, and anything with a side effect always miss. The similarity threshold is tuned on paraphrase pairs, and I log whether the answer came from exact, semantic, or live so I can catch wrong hits. I would not cache tool results that mutate state.”

Further reading (primary sources)

Related guides