Frontend Beauty
ESC

Type to search the knowledge base.

Streaming LLM Responses

Streaming is tokens over the wire so the UI can paint before the full answer exists. Backpressure, abort, and what to do with a partial JSON object.

intermediate6 min read
  • streaming
  • sse
  • ux

Users judge speed at the first token, not the last. A four-second spinner is a different product from a 200ms first byte and a paragraph that grows. LLM streaming is tokens (or events) over the wire so the UI can paint before the full answer exists.

It is still HTTP. It is still a sample. The new problems are partial data, abort, and backpressure. You cannot JSON.parse a half object. You must cancel the upstream when the user hits Stop. You must not setState on every byte if that locks the main thread.

Key takeaways

  • Time to first token is the UX metric. Total generation time is a second metric.
  • Streams are incremental events (SSE or a fetch ReadableStream). Buffer lines, not “the whole JSON.”
  • Abort with AbortController. Stopping the UI without canceling the server still bills you.
  • Partial JSON is not JSON. Wait for the object to close, or use a parser that yields incomplete trees on purpose.

First token is the product

Break latency into queue + prefill + decode. Prefill reads the prompt. Decode emits tokens one (or a few) at a time. Streaming hides decode behind paint. It does not hide a huge prefill. That is why prompt cache hits feel instant: prefill shrank. See time to first token.

A non-streaming JSON response waits for the last token, then downloads one blob. Fine for a 50-token classifier. Hostile for a 600-token explanation. Product choice per route, not a global SDK flag.

On the frontend this is the same instinct as streaming and Suspense: show something as soon as a chunk exists. The chunk here is a token delta, not an RSC payload.

LLM streaming on the wire

Two common shapes:

  1. SSE (text/event-stream): data: {json}\n\n frames, often ending with data: [DONE].
  2. Fetch body as a ReadableStream of bytes you decode yourself (or an SDK that already did).

The Vercel AI SDK’s streamText / UI message stream exists so you do not hand-roll frame parsing. You still need to know what a torn frame looks like.

async function streamAssistant(
  messages: { role: 'user' | 'assistant'; content: string }[],
  signal: AbortSignal,
  onDelta: (text: string) => void,
): Promise<string> {
  const res = await fetch('/api/chat', {
    method: 'POST',
    headers: { 'content-type': 'application/json', accept: 'text/event-stream' },
    body: JSON.stringify({ messages, stream: true }),
    signal,
  });

  if (!res.ok || !res.body) {
    throw new Error(`stream ${res.status}`);
  }

  const reader = res.body.getReader();
  const decoder = new TextDecoder();
  let buffer = '';
  let full = '';

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });

    const frames = buffer.split('\n\n');
    buffer = frames.pop() ?? '';

    for (const frame of frames) {
      const line = frame.split('\n').find((l) => l.startsWith('data: '));
      if (!line) continue;
      const data = line.slice(6);
      if (data === '[DONE]') return full;
      const json = JSON.parse(data) as { delta?: string };
      if (json.delta) {
        full += json.delta;
        onDelta(json.delta);
      }
    }
  }

  return full;
}

// Failure mode: JSON.parse(buffer) on the raw byte chunk.
// Frames tear across TCP packets. Incomplete JSON throws, you abort the UI.

Keep a string buffer. Split on the delimiter. Leave the incomplete tail in the buffer. Only parse a complete data: payload. That payload is one event, usually a small delta, not the assistant’s full JSON object.

Provider event names differ (content_block_delta, choices[0].delta.content, UI message chunks). Normalize on the server so the React tree sees onDelta.

Abort is a user action

Stop is not setState({ streaming: false }). The GPU will keep going. Use AbortController: pass signal into fetch, forward it to the vendor client on the server, and abort on unmount or on Stop.

let controller: AbortController | null = null;

function send(messages: { role: 'user' | 'assistant'; content: string }[]) {
  controller?.abort();
  controller = new AbortController();
  return streamAssistant(messages, controller.signal, paintDelta);
}

function stop() {
  controller?.abort();
  controller = null;
}

When fetch rejects with AbortError, do not toast “the model failed.” The user asked. Keep the partial text if it is prose. Discard it if you were streaming a structured object you cannot validate.

Double-send: disable the composer while streaming. A second fetch without aborting the first is two bills and a race on which bubble wins. Same bug as typeahead races.

Wire this through a route handler. The browser should not open an SSE connection to the model vendor with your key.

Partial JSON is not JSON

Prose streams are concatenation. Objects are not.

// After 40 tokens you might have:  {"vendorName":"Acme","tot
JSON.parse(partial); // SyntaxError every time until the last byte

Rules:

  1. Do not parse until finish / [DONE] if you need a valid object.
  2. If the UI wants fields early, use a streaming JSON parser or the SDK’s partial-object helper, and treat the result as Partial<T> until the final Zod parse succeeds.
  3. Never commit a partial to the database. A missing totalCents is not 0.
  4. Tool-call arguments also stream as strings. Wait for the tool-call event to close before JSON.parse of arguments.

Markdown in the bubble has the same class of bug: an unclosed ``` looks like a black hole. Render with a parser that tolerates incomplete input, or delay fence rendering until the close token.

Backpressure and the main thread

onDelta firing every token can mean hundreds of React renders. The event loop will not paint if you keep scheduling micro-work that never yields.

Batch:

function createPainter(flush: (text: string) => void) {
  let acc = '';
  let frame = 0;
  return (delta: string) => {
    acc += delta;
    if (frame) return;
    frame = requestAnimationFrame(() => {
      flush(acc);
      frame = 0;
    });
  };
}

If the consumer is slower than the producer (a phone rendering markdown), the ReadableStream buffers in memory. That is backpressure you are not applying. For long answers, prefer appending to a list of chunks and virtualizing the transcript (streaming chat UI) over concatenating one giant string that you re-parse every frame.

Do not block the reader loop on a slow await saveToIndexedDB(full) after every delta. Persist on DONE or on a throttle.

Errors mid-stream: you already painted text. Show a trailing error state on that bubble. Do not replace the bubble with a generic toast and pretend the tokens did not happen.

Out loud, in an interview

“Streaming is incremental tokens over SSE or a fetch ReadableStream so time to first token is what the user feels. I buffer frames, I never JSON.parse a torn chunk or a half object, and I abort with AbortController so Stop cancels the upstream. I batch paints on animation frames because a setState per token will hitch. Structured output waits for the stream to close before Zod runs.”

Further reading (primary sources)

Related guides