Frontend Beauty
ESC

Type to search the knowledge base.

Prompt Caching

Prompt caching reuses a stable prefix so you stop paying full price for the same system prompt and docs on every turn you send.

intermediate6 min read
  • caching
  • cost
  • latency

You send the same 8k-token style guide on every message. The model rereads it. Prompt caching is the first cost win that is not “use a smaller model.” A stable prefix (system prompt, tool schemas, a handbook) can be processed once and reused. Cached input is cheaper than uncached input. Prefill gets shorter. The window does not get bigger.

This is not HTTP caching and not a CDN. The cache key is the token prefix identity: the exact sequence, in order, plus the model and the bits of request config that change the rendered prompt. Bounce a timestamp into the first line and you miss every time.

Key takeaways

  • Hits require an identical prefix. Similar text does not count.
  • Put static content first: tools, system, long docs. Put the user turn last.
  • TTL is short (minutes unless you opt into a longer window). A miss is the default after idle.
  • Cached tokens still occupy the context window. You saved money and prefill, not capacity.

Prompt caching reuses the prefix you already send

Turn 1: system + tools + style guide + question. Turn 2: the same system + tools + style guide + history + next question. Without a cache, you pay full input price for the guide every time, and the model recomputes the prefix’s internal state (the KV cache, in serving terms).

Prompt caching keeps that prefix state around so the next request can skip the work. OpenAI applies this automatically on supported models when the prefix is long enough. Anthropic lets you opt in with cache_control (automatic on the last cacheable block, or explicit breakpoints on content blocks).

type CacheControl = { type: 'ephemeral'; ttl?: '5m' | '1h' };

const STYLE_GUIDE =
  'Answer from handbook passages. Cite ids. If the handbook is silent, say you do not know.';

const SEARCH_DOCS_TOOL = {
  name: 'search_docs',
  description: 'Search the product handbook',
  input_schema: {
    type: 'object',
    properties: { query: { type: 'string' } },
    required: ['query'],
  },
};

function docsAssistantBody(question: string, history: { role: string; content: string }[]) {
  const cache: CacheControl = { type: 'ephemeral' };
  return {
    model: process.env.LLM_MODEL,
    // Prefix order matters: tools, then system, then messages.
    tools: [SEARCH_DOCS_TOOL],
    system: [
      {
        type: 'text',
        text: STYLE_GUIDE, // large, versioned, rarely changes
        cache_control: cache,
      },
    ],
    messages: [...history, { role: 'user', content: question }],
  };
}

// Failure mode: STYLE_GUIDE + `new Date().toISOString()` in the same block.
// The prefix hash changes every call. You pay cache writes, never reads.

Read usage on the response. Providers split uncached input, cache-write input, and cache-read input. If writes stay high and reads stay zero, your prefix is moving.

This is closer to frontend caching strategies than it looks: identity of the key, TTL, and “what you store is the expensive computation.” It is unlike HTTP caching headers in one way: a 90% similar body is a miss. There is no ETag of “close enough.”

Prefix identity, not similar enough

A cache hit needs the rendered prefix to match. In practice that means:

You changed Cache
User question at the end Hit on the static prefix, if TTL is live
System prompt, tool order, or model id Miss from the first changed token
Header inserted above the docs Miss
Same prefix, new user turn Hit, if TTL still valid

OpenAI’s docs: put static content at the beginning, variable content at the end. Images, tools, and structured-output schemas sit in that prefix too. Anthropic’s docs: cache prefixes are created in order tools, system, then messages. Each level builds on the previous. A breakpoint writes a hash of everything up to that block. Changing an earlier block invalidates later ones.

Minimum prefix length exists. If your “style guide” is 200 tokens, you may never be eligible. Do not pad with junk to chase a hit. Either the prefix is naturally large (docs, tools, examples) or caching will not matter.

Where to put the breakpoint

Think in layers that change at different speeds.

  1. Tool schemas (deploy-scoped).
  2. System prompt + long reference docs (version-scoped).
  3. Conversation so far (turn-scoped, useful in multi-turn chats).
  4. This user question (never cache this as the end of a breakpoint if it includes a clock or nonce).

Anthropic allows a small number of explicit breakpoints. A common pattern is one on the system docs, and automatic caching that walks the growing conversation. Automatic mode that lands on a block which changes every request is a trap: you write a new entry every time and never read. Put the explicit breakpoint on the last stable block.

function splitForCache(params: {
  styleGuide: string;
  retrieved: string; // per question, do not bake into the cached block
  question: string;
}) {
  return {
    cachedPrefix: params.styleGuide,
    tail: `<retrieved>\n${params.retrieved}\n</retrieved>\n<question>${params.question}</question>`,
  };
}

Retrieved RAG chunks usually belong in the tail. They change with the question. If you retrieve a fixed corpus dump for every user of a product, that dump can live in the prefix. Most RAG apps should not.

Context engineering decides what is in the prefix. Caching only makes that prefix cheap.

TTL and miss reasons

Default lifetimes are short. Anthropic documents a five-minute ephemeral TTL, with a longer option that costs more to write. OpenAI’s automatic cache is also minutes-scale unless you opt into a longer window the vendor documents. Writes can cost more than a normal input token; reads cost less. Idle chats miss.

Lifetime is measured from the request that wrote or refreshed the entry. A long stream eats TTL, so a follow-up must start before the entry expires.

Miss reasons worth logging:

  • Prefix bytes changed (whitespace, JSON key order in tools).
  • Tool list filtered per user in a way that reorders schemas. Prefer allowed_tools / equivalent over deleting entries from the array, when the vendor supports it.
  • Clock, request id, or user id spliced into the system string. Put those in the tail, or in a later block without cache_control.
  • Load balancer / region: some systems hash the prefix to a machine. A prompt_cache_key (OpenAI) can help colocate related traffic. It does not override a changed prefix.

Watch hit rate in traces next to token counts. A feature that looks “free” in staging (one user, tight loops) can miss in production (sparse traffic).

What caching does not buy you

A larger window. Cached tokens still count. You cannot keep a 200k handbook in context for free attention.

Correctness or safety. A cached wrong policy is still wrong. Caching does not sanitize retrieved HTML. Version the docs in the prefix and bust on publish.

Semantic reuse. Matching “what’s the refund policy?” to yesterday’s answer is a semantic cache, a different product. Prompt caching is prefix identity only.

A substitute for trimming. Unbounded history still blows the window at a 100% hit rate on the system prompt. Pack first, cache second. After that, LLM cost optimization.

Out loud, in an interview

“Prompt caching reuses the KV state of a stable prefix so I stop paying full input price for the same system prompt, tools, and docs. Hits need byte-level prefix identity and a live TTL, so I put static content first and the user turn last. I never splice a timestamp into the cached block. Cached tokens still fill the context window. I would log cache reads versus writes and treat a write-only pattern as a breakpoint bug.”

Further reading (primary sources)

Related guides