Rate Limits from the Browser

Ten tabs and a keyboard shortcut will exhaust your own quota before a stranger does. Users mash Send, hold Enter, open the same chat in two windows, and bind “regenerate” to a key. An LLM rate limit is not only a provider header. It is a product surface: disable the composer, reject the extra POST, and say so.

Silent retries from the browser turn a 429 into a stampede. The tab that “just works” is the one that ate everyone else’s tokens.

Key takeaways

• Disable Send while a stream is live. That is courtesy, not a quota.
• Enforce a budget on the server per user or session. Tabs do not share React state.
• Distinguish your 429 from the provider’s 429. The UI copy and the retry policy differ.
• Idempotent send: the same client message id must not start two generations.
• Honor Retry-After. Do not spin. Do not hide the error in an infinite useEffect.

Two LLM rate limits, two owners

HTTP 429 means “too many requests.” It does not say whose ceiling you hit.

Source	Typical cause	What the user should see	Retry from the browser?
Your route	Session burst, daily cap, anonymous abuse	“Slow down” / “daily limit”	No, until the clock they can see
Provider	Org RPM/TPM, project cap	“The model is busy, try in a minute”	Only on the server, with jitter, for safe GETs and completions
Provider overload (often 503)	Model capacity, not your math	Same as busy	Server-side, bounded

OpenAI rate limits are org and project scoped, in requests and tokens per minute (and per day). They are not per browser tab. If five users share one key, they share one bucket. That is why client vs server LLM calls exists: you cannot attach a user to a key that lives in the bundle.

Your 429 is the one you can explain. “You sent 20 chats in a minute” is a better sentence than a raw provider error.

Disable the composer, then mean it

The first gate is UI state. While status is submitted or streaming, Send is a no-op. Stop is the only action. This is the same in-flight flag as a streaming chat UI.

'use client';

import { useChat } from '@ai-sdk/react';
import { DefaultChatTransport } from 'ai';
import { useEffect, useRef, useState } from 'react';

type RateLimitError = Error & { code: 'RATE_LIMIT'; retryAfterSec: number };

function isRateLimitError(err: unknown): err is RateLimitError {
  return err instanceof Error && (err as RateLimitError).code === 'RATE_LIMIT';
}

async function chatFetch(input: RequestInfo | URL, init?: RequestInit) {
  const res = await fetch(input, init);
  if (res.status !== 429) return res;
  const body = (await res.json().catch(() => ({}))) as {
    error?: string;
    retryAfterSec?: number;
  };
  const header = Number(res.headers.get('retry-after'));
  const retryAfterSec =
    body.retryAfterSec ?? (Number.isFinite(header) && header > 0 ? header : 60);
  const err = new Error(body.error ?? 'RATE_LIMIT') as RateLimitError;
  err.code = 'RATE_LIMIT';
  err.retryAfterSec = retryAfterSec;
  throw err;
}

export function GatedComposer() {
  const pendingId = useRef<string | null>(null);
  const [lockedUntil, setLockedUntil] = useState<number | null>(null);
  const { sendMessage, status, stop, error } = useChat({
    transport: new DefaultChatTransport({ api: '/api/chat', fetch: chatFetch }),
    onError(err) {
      if (isRateLimitError(err)) {
        setLockedUntil(Date.now() + err.retryAfterSec * 1000);
      }
    },
  });
  const [input, setInput] = useState('');
  const inFlight = status === 'submitted' || status === 'streaming';
  const locked = lockedUntil != null;

  useEffect(() => {
    if (status === 'ready' || status === 'error') pendingId.current = null;
  }, [status]);

  useEffect(() => {
    if (lockedUntil == null) return;
    const ms = lockedUntil - Date.now();
    if (ms <= 0) {
      setLockedUntil(null);
      return;
    }
    const t = window.setTimeout(() => setLockedUntil(null), ms);
    return () => window.clearTimeout(t);
  }, [lockedUntil]);

  function onSubmit(e: React.FormEvent) {
    e.preventDefault();
    const text = input.trim();
    if (!text || inFlight || locked) return;
    // Reuse the id if both clicks land before `status` flips. Two UUIDs would
    // bill twice; the server collapses on this one field.
    const clientMsgId = pendingId.current ?? crypto.randomUUID();
    pendingId.current = clientMsgId;
    sendMessage(
      { text },
      {
        headers: { 'x-client-msg-id': clientMsgId },
        body: { clientMsgId },
      },
    );
    setInput('');
  }

  return (
    <form onSubmit={onSubmit}>
      <textarea
        value={input}
        onChange={(e) => setInput(e.target.value)}
        onKeyDown={(e) => {
          if (e.key === 'Enter' && !e.shiftKey) {
            e.preventDefault();
            if (!inFlight && !locked) e.currentTarget.form?.requestSubmit();
          }
        }}
      />
      {inFlight ? (
        <button type="button" onClick={() => stop()}>
          Stop
        </button>
      ) : (
        <button type="submit" disabled={locked}>
          Send
        </button>
      )}
      {isRateLimitError(error) ? (
        <p role="alert">You have hit this hour's limit. Try later.</p>
      ) : null}
    </form>
  );
}

This stops double-submit in one tab. It does nothing for the other nine. Server quota is the real gate.

Default useChat / DefaultChatTransport POST JSON is { id, messages, trigger, messageId } plus extra body fields from the second sendMessage argument. sendMessage({ text, metadata: { clientMsgId } }) puts the key on the last UIMessage, not at the top level. The Route Handler below reads clientMsgId from that extra body, then the x-client-msg-id header. Message metadata is the wrong place if the server never walks messages.

useChat will not turn a 429 JSON body into error.message === 'RATE_LIMIT'. The server’s throw never leaves the isolate. Parse the 429 in a custom fetch (or a custom transport) and set a code the composer can branch on.

Server quota per session

Count completed and in-flight generations. A stream that has not finished still spends.

// lib/quota.ts
// Do not use a process-local Map on Vercel. Isolates do not share it.
import { Redis } from '@upstash/redis';

const redis = Redis.fromEnv();
const WINDOW_SEC = 60;
const MAX_PER_WINDOW = 8;

export async function assertChatQuota(userId: string) {
  const key = `chat-quota:${userId}`;
  const count = await redis.incr(key);
  if (count === 1) await redis.expire(key, WINDOW_SEC);
  if (count <= MAX_PER_WINDOW) return;

  const ttl = await redis.ttl(key);
  const err = new Error('RATE_LIMIT') as Error & {
    code: 'RATE_LIMIT';
    retryAfterSec: number;
  };
  err.code = 'RATE_LIMIT';
  err.retryAfterSec = ttl > 0 ? ttl : WINDOW_SEC;
  throw err;
}

// app/api/chat/route.ts
import { getSession } from '@/lib/auth';
import { alreadyProcessed } from '@/lib/idempotency';
import { assertChatQuota } from '@/lib/quota';

export async function POST(req: Request) {
  const session = await getSession(req);
  if (!session) return new Response('Unauthorized', { status: 401 });

  const payload = (await req.json()) as {
    id?: string;
    messages?: unknown[];
    trigger?: string;
    messageId?: string;
    clientMsgId?: unknown;
  };
  const clientMsgId =
    typeof payload.clientMsgId === 'string'
      ? payload.clientMsgId
      : req.headers.get('x-client-msg-id');
  if (!clientMsgId) {
    return new Response(JSON.stringify({ error: 'MISSING_CLIENT_MSG_ID' }), {
      status: 400,
      headers: { 'content-type': 'application/json' },
    });
  }
  if (await alreadyProcessed(session.userId, clientMsgId)) {
    return new Response('Duplicate', { status: 409 });
  }

  try {
    // Count at accept, before streamText. Stop still spent a slot.
    await assertChatQuota(session.userId);
  } catch (err) {
    const retry = (err as { retryAfterSec?: number }).retryAfterSec ?? 60;
    return new Response(JSON.stringify({ error: 'RATE_LIMIT', retryAfterSec: retry }), {
      status: 429,
      headers: {
        'content-type': 'application/json',
        'retry-after': String(retry),
      },
    });
  }

  // streamText …
}

Idempotency: if the user double-clicks and both POSTs leave before React sets busy, the same clientMsgId must collapse to one generation. Store it next to the conversation id. A 409 is better than two assistant rows and two bills.

Anonymous users get a session cookie, not “no limit.” IP limits are a backstop. They punish coffee-shop NAT.

What to show, what to retry

Provider 429s belong in the Route Handler, not in useEffect. The server may retry a completion with jitter after reading Retry-After or x-ratelimit-reset-requests. It must not retry a tool that already refunded a card. That split is retries, timeouts, and fallbacks.

The browser:

1. Reads 429 (the custom fetch above parses the JSON and Retry-After).
2. Renders the message from error.code, not from a thrown server Error.
3. Disables Send until retryAfterSec elapses, or shows a countdown.
4. Does not loop sendMessage on a timer.

Copy that names the owner:

• Your cap: “You have used this hour’s chats.”
• Provider cap: “The model provider is throttling us. We are on it.”
• Do not say “error 429” as the only string.

Abort in-flight work when the user hits Stop. The quota sample counts at accept, so Stop still spent a slot. That discourages mash-Stop-mash. If you refund on abort-before-first-token, decrement the same Redis key. Do not skip the increment.

Footguns

• Retrying 429 in the client with exponential backoff and no jitter. Ten tabs synchronize and hit you on the same second.
• Only rate-limiting POST /api/chat and leaving /api/embed open. People will hammer search.
• Using one global in-memory Map on serverless. Each isolate has its own map. Use Redis or the provider you already have for sessions.
• Hiding the 429 behind a generic toast that auto-retries. That is how you DDoS yourself.
• Trusting disabled={isLoading} without a server check. View source is not required to curl.

Interview answer (out loud)

“Users will hammer Send across tabs, so the composer disable is UX and the quota is server-side per user. I send a client message id for idempotency. My 429 is a product limit with Retry-After. The provider 429 is handled on the server with jitter, never in a client loop, and never for a successful write. The UI names which limit fired.”

Further reading (primary sources)

• OpenAI rate limits
• MDN 429 Too Many Requests
• AbortController

Related on this site

• Retries, timeouts, and fallbacks
• Client vs server LLM calls
• Streaming chat UI in React
• AbortController
• Route Handlers

Related guides

• LLMOps LifecycleLLMOps is how an LLM feature is versioned, evaluated, deployed, and watched. Prompts are artifacts. Traces are tests waiting to happen.
• Retries, Timeouts, and FallbacksLLM APIs flake. Retry with jitter on 429/5xx, bound the wait, fall back to a smaller model, and never retry a successful side-effecting tool.
• Streaming LLM ResponsesStreaming is tokens over the wire so the UI can paint before the full answer exists. Backpressure, abort, and what to do with a partial JSON object.
• Time to First Token and LatencyUsers feel TTFT. Total time is TTFT plus generation. Streaming, smaller models, and prompt cache hit rate are the knobs that actually move the UI.
• Agent Failure ModesAgents loop, lie about tool results, skip confirmation, and get stuck apologizing. Catalog the failures and the circuit breakers.