Frontend Beauty
ESC

Type to search the knowledge base.

Speculative Decoding

Speculative decoding drafts tokens with a cheap model and verifies them with the big one. Why checking a draft can beat writing every token the hard way.

advanced6 min read
  • inference
  • latency
  • decoding

Prefill can chew the prompt in parallel. Decode cannot: each token waits on the one before it. That sequential loop is why a long answer feels sticky even on a huge GPU. Speculative decoding (also called assisted generation) steals parallelism back. A cheap drafter proposes several tokens. The large model checks them in one forward pass. If the guesses were good, you keep a block of tokens for roughly the cost of one large-model step.

You almost never implement this yourself in a TypeScript app. You consume it when a provider or a serving stack (vLLM, TGI, TensorRT-LLM) turns it on. You still need the intuition, because acceptance rate is why one workload speeds up and another does not.

Key takeaways

  • Autoregressive decode is sequential and usually memory-bound. Prefill is the parallel part.
  • Draft-then-verify: a small model (or n-gram / extra heads) proposes k tokens; the target model accepts a prefix.
  • With a proper accept/reject rule, the output distribution matches the large model. It is a speed trick, not a different personality.
  • If drafts are rarely accepted, you paid for extra work and can go slower. Acceptance rate is the metric.
  • App engineers enable this via a serving flag or a provider. Do not bolt a second fetch onto your chat route and call it speculation.

Why decode is the bottleneck

A transformer step for one new token reuses the KV cache for the prefix, then spends most of its time moving weights. Compute sits idle. You cannot batch “the next token” with “the token after that” for the same sequence, because you do not know it yet.

Prefill is different: all prompt tokens are known, so the model can process them together. That is why a 4k-token system prompt has a visible TTFT tax, and why a cache hit on that prefix makes the blank box shorter. Speculative decoding does not remove prefill. It attacks the sequential tail.

Speculative decoding: draft, then verify

The control flow, stripped of CUDA:

  1. Drafter produces k candidate tokens given the current prefix.
  2. Target model runs one forward pass that scores those k positions (plus the following token).
  3. Walk left to right. Keep a candidate if the target would have sampled it (or passes a rejection test that preserves the target distribution). Stop at the first rejection.
  4. If you rejected, sample the replacement from the target. If you accepted all k, sample one extra token from the leftover target distribution.
  5. Append accepted tokens. Repeat.
type Sampler = {
  draft: (prefix: number[], k: number) => number[];
  // One target pass that returns a next-token distribution at each draft position.
  targetVerify: (prefix: number[], draft: number[]) => number[][];
  sample: (probs: number[]) => number;
  accept: (draftToken: number, targetProbs: number[]) => boolean;
};

function speculativeBlock(prefix: number[], s: Sampler, k: number): number[] {
  const drafted = s.draft(prefix, k);
  const dists = s.targetVerify(prefix, drafted);
  const accepted: number[] = [];

  for (let i = 0; i < drafted.length; i++) {
    if (s.accept(drafted[i], dists[i])) {
      accepted.push(drafted[i]);
    } else {
      accepted.push(s.sample(dists[i]));
      return accepted;
    }
  }

  // All k matched; take a bonus token from the leftover distribution.
  accepted.push(s.sample(dists[k] ?? dists[k - 1]));
  return accepted;
}

This is the shape, not a production sampler. Real systems use the rejection rule from the speculative decoding papers so accepted tokens are distributed like the target, not like the drafter. Greedy “keep if top-1 matches” is a common approximation; it can change outputs.

Hugging Face TGI’s speculation docs describe the same idea under a few names: speculative decoding, assisted generation, Medusa. TGI ships Medusa (extra heads on the target that guess several tokens) and n-gram (copy a continuation that already appeared in the context). N-gram helps most on repetitive text and code. It can hurt when it misses.

Acceptance rate is the whole game

Speedup exists only when the target agrees with the draft often enough to amortize the extra work.

Drafter Tends to work when Tends to fail when
Smaller model in the same family Similar style, low temperature Creative sampling, domain shift
Medusa / extra heads You trained the heads for this target Off-the-shelf target with no heads
N-gram / prompt lookup Code, HTML, copied identifiers Open-ended chat with no repeats

Temperature and top-p change this. High entropy sampling makes the next token less predictable, so drafts miss more. Structured JSON and code, especially with a schema, often accept longer blocks. That is why serving people talk about speculation helping coding models more than poetry.

If you self-host, log:

  • mean accepted draft length
  • fraction of blocks with zero accepts
  • tokens/sec with speculation on vs off, on your prompt mix

A blog-post speedup on a different corpus is not your number. Do not copy it into a slide.

When drafts hurt, and who should flip the switch

Drafting is extra compute and extra memory. You lose when:

  • The drafter is not actually cheap (wrong GPU split, huge “small” model).
  • Acceptance collapses (high temperature, a drafter from another family).
  • Batch throughput matters more than single-stream latency, and speculation wrecks batching on your stack.
  • You draft k too large. Long drafts that die at token 1 are wasted verify width.

For an app on a hosted API, you usually cannot pick k. You pick a model id that the provider already serves with speculation, or you do not. Predicted outputs (where you know most of the text, like a code edit) are a cousin: you hand the server a guess, it verifies. That is worth using when the UI is “edit this file,” not when the answer is unknown.

Do not implement DIY speculation as two chained chat completions in Next.js. The verify pass has to be a single target forward over the draft, inside the serving engine. Two HTTP generations will add round trips and can change the distribution.

Time to first token still depends on prefill. Speculation helps the tokens after that. Stream either way.

Interview answer (out loud)

“Speculative decoding drafts k tokens with a cheap model or extra heads, then the large model verifies that block in one pass. Accepted prefixes keep the target distribution if you use the right rejection rule, so quality should match the large model. The speedup is acceptance rate times how expensive decode was. I would enable it in the serving stack and measure on our traces. I would not fake it with two API calls from the app.”

Further reading (primary sources)

Related guides