Frontend Beauty
ESC

Type to search the knowledge base.

Time to First Token and Latency

Users feel TTFT. Total time is TTFT plus generation. Streaming, smaller models, and prompt cache hit rate are the knobs that actually move the UI.

intermediate6 min read
  • latency
  • streaming
  • ux

A nine-second blank box is a bounce. Tokens per second after that is a number for a benchmark graph. Time to first token (TTFT) is when the UI can paint. Total latency is TTFT plus the rest of generation. If you only log “request duration,” you will optimize the wrong half.

Frontend engineers already know this from TTFB and streaming HTML. Same product: show something, keep the input usable, let the user abort.

Key takeaways

  • TTFT is queue + prefill (and your own retrieval). Decode time is mostly output tokens / tokens per second.
  • Users judge the blank state. Stream. Measure p95 TTFT, not the mean of happy-path demos.
  • Smaller models, prompt cache hits, and fewer input tokens move TTFT. Verbosity and max tokens move time-to-last-token.
  • Cancel in-flight work with AbortController. A discarded stream still bills if you do not close it.
  • Disable Send while streaming. Stacked requests make every latency graph a lie.

Time to first token is what users feel

ChatGPT-shaped UIs trained people to expect a caret. A spinner for the whole answer feels like search in 1998. Stream tokens (or at least the first sentence) and the same total time feels faster because reading overlaps decode.

Total time still matters for JSON that the UI cannot render until the object closes. For those calls, TTFT is hidden; time-to-valid-object is the metric. Do not stream a broken prefix into JSON.parse. See streaming LLM responses.

Queue, prefill, decode

Break a completion into stages you can actually attribute:

Stage What it is Moves with
Your work Auth, retrieval, prompt pack Index, network, chunk count
Provider queue Waiting for a slot Load, tier, region
Prefill Parallel ingest of the prompt Input tokens, cache hit, model size
Decode One (or a few) tokens at a time Output tokens, speculative decoding, model size

TTFT ≈ your work + queue + prefill. Time to last token ≈ TTFT + decode. Prompt cache hits make prefill cheaper and often faster because the prefix’s KV state is reused. Speculative decoding mostly helps decode, not the blank box, unless the first tokens are drafted too.

type Latency = {
  retrieveMs: number;
  ttftMs: number | null;
  totalMs: number;
  outputTokens: number;
};

type StreamChunk = { text?: string; usage?: { outputTokens: number } };

async function timedCompletion(
  packPrompt: () => Promise<CompletionRequest>,
  stream: (req: CompletionRequest, signal: AbortSignal) => AsyncIterable<StreamChunk>,
  signal: AbortSignal,
): Promise<{ text: string; latency: Latency }> {
  const t0 = performance.now();
  const req = await packPrompt();
  const retrieveMs = performance.now() - t0;

  let ttftMs: number | null = null;
  let text = '';
  let outputTokens = 0;

  for await (const chunk of stream(req, signal)) {
    if (chunk.text) {
      if (ttftMs === null) ttftMs = performance.now() - t0;
      text += chunk.text;
    }
    if (chunk.usage) outputTokens = chunk.usage.outputTokens;
  }

  return {
    text,
    latency: {
      retrieveMs,
      ttftMs,
      totalMs: performance.now() - t0,
      outputTokens,
    },
  };
}

Put ttft_ms and retrieve_ms on the generation span. If p95 TTFT is 8s and retrieve is 7s, the model is not your problem.

Measure p95, not the happy path

Mean TTFT on an empty staging index is fan fiction. Track:

  1. p50 / p95 / p99 TTFT by route and model.
  2. p95 time to last token for non-streamed structured calls.
  3. Abort rate and time-to-abort. Users who hit Stop at 6s are telling you the blank box failed.
  4. Cache hit rate next to TTFT. A miss that used to be a hit will show up here first.

Pin model ids in the dashboard. An alias that silently moved you to a bigger model will look like “the app got slow.”

OpenAI’s own latency guide is blunt: generate fewer tokens, use a faster model, stream so people wait less. Cutting output tokens often cuts total time almost in proportion, because decode is sequential. Cutting input tokens helps TTFT when prefill is the long pole: huge prompts, images, cache misses. On a cache hit with a short suffix, input size barely moves the blank box.

Frontend knobs that actually move the UI

  • Stream to the list, not through global React state on every token. Update the in-flight message; do not reconcile the whole transcript. See streaming chat UI.
  • AbortController on Stop and on navigate away. Tie it to the same signal you pass the route handler. See AbortController.
  • Disable the composer while in flight. Rate limits and stacked prefills are self-inflicted. Surface 429s instead of silent retry storms.
  • Optimistic user bubble immediately; keep a skeleton on the assistant side until first token, then replace.
  • Do not block the input on last token if you can allow queueing one follow-up. Blocking forever feels broken; unlimited queue feels like a denial-of-service.
import { useRef } from 'react';

function useChatAbort() {
  const ref = useRef<AbortController | null>(null);

  function start() {
    ref.current?.abort();
    const next = new AbortController();
    ref.current = next;
    return next.signal;
  }

  function stop() {
    ref.current?.abort();
  }

  return { start, stop };
}

useRef keeps the in-flight controller across re-renders. A plain { current: new AbortController() } inside the function is a new object every call, so stop() cannot abort the request that start() already handed to fetch.

If you use Next.js streaming UI, the same abort story applies across Suspense boundaries: a leftover server stream is still work you pay for.

Footguns

  • Measuring only total duration on streamed endpoints. The connection stays open for the whole decode; TTFT is what users felt.
  • Prefetching a completion on hover. You will pay for curiosity.
  • Giant tool schemas in the prefix that miss cache because you reorder tools per user.
  • Waiting for moderation of the full answer before first paint. Moderate incrementally or on the prompt, or you threw away streaming.
  • Showing a typing indicator that lies (animating while you are still retrieving). Be honest: “Searching docs…” then stream.

Interview answer (out loud)

“Time to first token is queue plus prefill plus my retrieval. Total time adds decode, which tracks output tokens. I stream so the blank box is short, log p95 TTFT separately from time-to-last-token, and cancel with AbortController. The knobs that move TTFT are a smaller model, a cache hit on a stable prefix, and doing less work before the first generate. Tokens per second is not the product metric.”

Further reading (primary sources)

Related guides