Frontend Beauty
ESC

Type to search the knowledge base.

KV Cache and Inference Basics

Autoregressive decoding reuses key/value tensors from earlier tokens. That is why a long prompt has a prefill cost and why cache hits feel instant.

advanced7 min read
  • inference
  • kv-cache
  • performance

Turn two of a chat feels cheaper than turn one even though you sent the whole history again. That is not the model “remembering you.” It is the KV cache: the serving stack kept the key and value tensors from the shared prefix and only ran the new tokens. Prompt caching products are this idea with a bill attached.

You do not need CUDA to ship a chat app. You do need the prefill vs decode split, because it explains time to first token, why long answers stall, and why shuffling the system prompt wrecks cache hit rate.

Key takeaways

  • Autoregressive models emit one token at a time. Each step attends over everything before it.
  • The KV cache stores per-layer keys and values so those projections are not recomputed.
  • Prefill processes the prompt (often in parallel). Decode is sequential and usually dominates user-visible latency.
  • Stable prefixes make provider prompt caches work. User text belongs at the end.
  • Batching raises GPU utilization and can raise your p95. Throughput is not the same as UX.

What the KV cache stores

Attention needs, for each new token, a query against the keys of previous tokens, then a mix of their values. Without a cache, every decode step would reproject the entire prefix. With a cache, you project only the new token, append its K and V, and attend.

Hugging Face’s cache docs put it bluntly: without caching, attention cost per step grows with the whole sequence; with caching you compute the current K/V and reuse the rest. Memory grows with sequence length × layers × heads × head dim. That memory is why long-context serving is expensive even when the user asked a short question.

Prefill vs decode, as a procedure:

  1. Prefill. For each prompt token, project K and V once. Store them in order. This pass is parallel and sets time to first token.
  2. Sample the first output token from that pass.
  3. Decode. Project K and V for only the new token. Attend against the stored prefix. Append the new K/V. Repeat until stop.
  4. Never reproject the prefix. That is the whole cache. A naive loop reprojects tokens 1..n at step n.
type Kv = { k: number; v: number };

function project(tokenId: number): Kv {
  // Stand-in for W_k / W_v. Real tensors are per layer and head.
  return { k: tokenId * 2, v: tokenId * 3 };
}

function prefill(prompt: number[]): { cache: Kv[]; projections: number } {
  const cache = prompt.map(project);
  return { cache, projections: prompt.length };
}

function decodeOne(cache: Kv[], newToken: number): number {
  cache.push(project(newToken));
  return 1; // projections this step: the new token only
}

function decodeNaive(allTokens: number[]): number {
  let projections = 0;
  for (let i = 0; i < allTokens.length; i++) {
    for (let j = 0; j <= i; j++) {
      project(allTokens[j]);
      projections += 1;
    }
  }
  return projections;
}

export function compareWork(prompt: number[], newTokens: number[]) {
  const { cache, projections: prefillCost } = prefill(prompt);
  let decodeCost = 0;
  for (const t of newTokens) decodeCost += decodeOne(cache, t);
  return {
    cachedProjections: prefillCost + decodeCost, // 4 + 1 + 1 = 6
    naiveProjections: decodeNaive([...prompt, ...newTokens]), // 21
    cacheLen: cache.length,
  };
}

compareWork([1, 2, 3, 4], [5, 6]);

The numbers are counters, not CUDA. They show why turn two is cheaper when the prefix is reused, and why an 8k-token style guide on every turn is a prefill bill unless that prefix is identical and still in cache. Your app only sees HTTP. The serving stack is doing prefill then decodeOne.

Prefill, then decode

Phase What the GPU does What the user feels
Queue Wait for a slot Blank box
Prefill Forward over the prompt, fill KV Time to first token
Decode One token per step, append KV Tokens per second
Stop EOS, max tokens, or abort Final bubble

OpenAI’s latency guide notes the awkward split: cutting output tokens often cuts latency almost in proportion, because decode is sequential. Cutting input tokens helps TTFT when prefill is the long pole: huge prompts, images, cache misses. A cache hit on a huge prompt is why turn two can feel instant: prefill of the prefix was skipped or heavily reduced. You still prefill the uncached suffix.

type Budget = {
  promptTokens: number;
  suffixTokens: number;
  maxNewTokens: number;
  cacheHit: boolean;
};

export function roughWork(b: Budget) {
  return {
    prefillTokens: b.cacheHit ? b.suffixTokens : b.promptTokens,
    decodeSteps: b.maxNewTokens,
  };
}

// App-level: cap maxNewTokens for "yes/no" tools.
// Do not raise max tokens "just in case" and then wonder why p95 died.

Streaming does not make decode faster. It makes the wait honest. See time to first token.

What prompt caching is actually doing

Provider prompt caching is KV reuse with an API and a TTL. The common rules from the outside:

  • Byte-identical prefix. A changed system prompt, a reordered tool list, or a timestamp in the first message is a miss.
  • Put static content first. Policy, tool schemas, the handbook. Dynamic user text and retrieved chunks go last so the cached prefix is long.
  • TTL is not your RAM. A miss after idle is normal. Do not write a comment that says “the model remembers.”
  • Cached input is cheaper than uncached input. How much cheaper changes. Do not bake a dollar figure into a dashboard title.

If you bounce the prefix, you paid prefill again and you evicted someone else’s useful cache line on a busy replica. That is an app bug, not a GPU mystery.

Speculative decoding is a different trick (draft, then verify). It is not KV reuse.

Batching: throughput vs your p95

A GPU likes fat matrix multiplies. One user’s single-token decode is a skinny job. Serving stacks batch many sequences so prefill and decode share the chip (continuous batching). That raises tokens/sec for the box and can add queueing for a person.

As an application engineer:

  • Do not assume a bigger max_tokens is free because the GPU is “already on.”
  • Do not issue 12 parallel agent steps if one sequential tool would do. You are competing with yourself for batch slots.
  • Do abort. A closed tab that keeps decoding still occupies KV memory until the server notices.
  • Measure p95 TTFT and p95 total, not only average tokens/sec from a blog.

Local toy servers that disable the KV cache (or recompute on every token) will feel inexplicably slow. Production APIs keep it on. You still pay for the memory: very long conversations get truncated or summarized because the cache cannot grow forever.

What you can control from the app

You will not pick StaticCache vs QuantizedCache on a hosted API. You can still:

  1. Stabilize the prefix. Version the system prompt. Do not interpolate new Date().toISOString() at the top.
  2. Shorten decode. Ask for JSON, not an essay. Lower max_tokens for tools.
  3. Avoid rewriting history. Re-sending a paraphrased summary every turn can destroy prefix identity.
  4. Count images and retrieved chapters as prefill, not as “free context.”
  5. Trace cache hits if the provider returns them. A sudden miss rate is a prompt-ordering bug.

This is enough inference literacy to talk to a platform team without pretending you wrote a kernel.

Interview answer (out loud)

“The KV cache stores per-layer key and value tensors from tokens already processed so decode does not reproject the whole prefix. Prefill fills that cache for the prompt and sets time to first token. Decode is sequential, one token per step, which is why long answers feel slow. Provider prompt caches reuse a stable prefix; I put static instructions first and user text last. Batching helps throughput and can hurt my p95, so I cap max tokens, abort unused streams, and I do not shuffle the system prompt.”

Further reading (primary sources)

Related guides