Frontend Beauty
ESC

Type to search the knowledge base.

Conversation State in Chat UIs

Chat conversation state is messages, in-flight status, and a conversation id. Persist on the server, trim for the model, cache in the UI.

intermediate7 min read
  • react
  • state
  • chat

localStorage with 80 turns of markdown will freeze Safari on restore and still miss the tool results you needed for the next call. Chat conversation state is not “the array I useState.” It is a conversation id, a server-owned transcript, a trimmed view for the model, and a small client cache for the glass.

Embeddings do not belong in React state. Neither does the full retrieval corpus. If you cannot refresh the tab and land on the same thread, you do not have a product. You have a demo.

Key takeaways

  • The server owns the thread. The client caches a window of UI messages.
  • Give every conversation an id before the first token. Send it on every request.
  • Persist UI-shaped messages (including tool parts). Convert to model messages at the edge.
  • Trim and summarize for the model. Do not trim the user’s scrollback without a “load earlier” path.
  • In-flight status is local. Do not persist streaming across a reload as if the generator were still running unless you can resume.

Chat conversation state is three stores

Thinking in React still applies: find the state, find the owner.

Store Holds Lifetime
Server thread Canonical messages, user id, created at, title Until the user deletes it
Model window Subset (or summary + tail) you actually send Per request
Client cache What the list renders, plus status Tab, plus optional disk for instant paint

Mixing them is how you upload a 2 MB JSON blob on every keystroke of Send, or how you drop tool results on refresh and the model “forgets” it already refunded.

The AI SDK’s useChat({ id, messages }) is the client cache. Persistence is your database. See their chatbot docs for the hook. This page is the ownership split.

The conversation id is the product

Mint the id when the user opens a new chat, not when the first assistant token returns. Put it in the URL (/chat/[id]) so refresh, share, and multi-tab have a name.

// app/api/chat/route.ts
import { streamText, convertToModelMessages, type UIMessage } from 'ai';
import { getSession } from '@/lib/auth';
import { loadThread, appendMessages, mergeTail, trimForModel } from '@/lib/threads';

function countMessageTokens(m: { content?: unknown }): number {
  // Schematic ceiling (chars/4). Swap in the model's tokenizer in production.
  return Math.ceil(JSON.stringify(m.content ?? '').length / 4);
}

export async function POST(req: Request) {
  const session = await getSession(req);
  if (!session) return new Response('Unauthorized', { status: 401 });
  // DefaultChatTransport sends `id`. A custom prepare hook below sends `message`.
  const { id, message }: { id?: string; message?: UIMessage } = await req.json();
  if (!id || !message) return new Response('Bad request', { status: 400 });

  const thread = await loadThread(session.userId, id);
  if (!thread) return new Response('Not found', { status: 404 });

  // Persist the user turn before the model runs. Reload mid-stream keeps the question.
  await appendMessages(id, [message]);

  const uiMessages = mergeTail(thread.messages, [message]);
  const modelMessages = trimForModel(
    await convertToModelMessages(uiMessages),
    countMessageTokens,
    8_000, // tokens, not characters
  );

  const result = streamText({
    model: 'openai/gpt-4.1-mini',
    messages: modelMessages,
    abortSignal: req.signal,
    async onFinish({ response }) {
      // Runs on abort too. Write whatever assistant parts you have.
      await appendMessages(id, response.messages);
    },
  });

  return result.toUIMessageStreamResponse();
}

mergeTail exists because the row you just accepted is not in the loadThread snapshot. Authenticate the conversation id. A guessed UUID must not load someone else’s thread. The field name is id on both sides: that is what useChat({ id }) puts on the wire unless you rename it in prepareSendMessagesRequest.

On the client:

'use client';

import { useChat } from '@ai-sdk/react';
import { DefaultChatTransport } from 'ai';
import type { UIMessage } from 'ai';

export function Thread({
  id,
  initialMessages,
}: {
  id: string;
  initialMessages: UIMessage[];
}) {
  const { messages, sendMessage, status } = useChat({
    id,
    messages: initialMessages,
    transport: new DefaultChatTransport({
      api: '/api/chat',
      // Default body is { id, messages, trigger, messageId }. Send the tail only.
      prepareSendMessagesRequest: ({ id: chatId, messages: all }) => ({
        body: { id: chatId, message: all.at(-1) },
      }),
    }),
  });

  // status is local. initialMessages came from the server.
  return (
    <ChatList
      messages={messages}
      status={status}
      onSend={(text) => sendMessage({ text })}
    />
  );
}

Load initialMessages in a Server Component. Do not fetch the whole history in useEffect after paint if you can avoid the flash. useEffect is for syncing, not for being your database.

What the model sees vs what the UI shows

The UI wants every user bubble, every tool chip, every citation. The model wants a budget. Context engineering is the discipline. The chat-specific rule:

export function mergeTail<T extends { id: string }>(saved: T[], clientTail: T[]): T[] {
  const ids = new Set(saved.map((m) => m.id));
  return [...saved, ...clientTail.filter((m) => !ids.has(m.id))];
}

export function trimForModel<T extends { role: string }>(
  messages: T[],
  countTokens: (m: T) => number,
  budgetTokens = 8_000,
): T[] {
  const system = messages.filter((m) => m.role === 'system');
  const rest = messages.filter((m) => m.role !== 'system');
  const kept: T[] = [];
  let used = system.reduce((n, m) => n + countTokens(m), 0);

  for (let i = rest.length - 1; i >= 0; i--) {
    const cost = countTokens(rest[i]);
    if (used + cost > budgetTokens && kept.length > 0) break;
    kept.push(rest[i]);
    used += cost;
  }
  return [...system, ...kept.reverse()];
}

countTokens is required. The default budget is 8_000 tokens. The route passes countMessageTokens, a schematic chars/4 ceiling. Swap in the model’s tokenizer in production. Dropping from the front (oldest) is the default. Dropping tool results from the middle is how the model repeats a call. If you strip tools, replace them with a short summary: “Used getOrder; status is shipped.”

A rolling summary is a message you write, stored on the thread, not a vibe:

async function compact(threadId: string) {
  const old = await loadOldTurns(threadId);
  const { text } = await generateText({
    model: 'openai/gpt-4.1-mini',
    prompt: `Summarize these turns for future context. Keep ids and decisions.\n${format(old)}`,
  });
  await writeSummary(threadId, text);
  await markCompacted(threadId, old.at(-1)!.id);
}

That summary is agent memory in the small: working window vs long-term blob. Do not embed each turn into React state “so we can search locally.” Search is a server index.

Persistence that does not freeze the tab

  • Write the user message as soon as you accept it. Reload mid-stream should show the question. The partial assistant answer is optional.
  • Store UI messages, not only concatenated strings. Tool parts, ids, and roles are how you reconstruct streaming chat.
  • Disk cache, if any, is a snapshot. IndexedDB beats localStorage for large transcripts. Quota and JSON parse cost on the main thread are why Safari hitch-steps. See the event loop: a 2 MB JSON.parse in useEffect is a long task.
  • Never persist secrets the user pasted. Redact API keys on the way in.

Resume is a special case. If the server can continue a stream (same conversation id, still generating), the client may reconnect. If it cannot, mark the assistant row complete-or-partial and let the user regenerate. Do not leave status: 'streaming' in the database overnight.

Footguns

  • Sending the entire thread from the client as the only copy. The client will drop parts, and a modified client will inject tool results.
  • Using array indexes as message ids. Delete and regenerate will re-key the list.
  • Putting vector rows in context. You will serialize embeddings into the chat JSON. Retrieval belongs in the route, per turn.
  • One React context for all conversations. Switching threads mid-stream races setMessages. Key the chat component by id so it remounts.
  • Summarizing without versioning. A bad summary poisons every later turn. Keep the raw tail.

Interview answer (out loud)

“Conversation state is a server-owned thread keyed by id, a trimmed model window, and a client cache of UI messages plus in-flight status. I persist UI-shaped messages, including tool parts, and convert to model messages on the server. I trim from the oldest turns or replace them with a stored summary. I do not keep embeddings in React state, and I do not treat localStorage as the source of truth.”

Further reading (primary sources)

Related guides