Frontend Beauty
ESC

Type to search the knowledge base.

Retries, Timeouts, and Fallbacks

LLM APIs flake. Retry with jitter on 429/5xx, bound the wait, fall back to a smaller model, and never retry a successful side-effecting tool.

intermediate7 min read
  • reliability
  • retries
  • production

A 503 on complete() is annoying. A 503 on charge_card is an incident. LLM retries look like ordinary fetch retries until a tool inside the same request already moved money, sent email, or created a ticket. Retrying generation is how you survive a blip. Retrying a side effect is how you double-bill.

Users mash Stop while your server is still waiting on a provider that woke up late. If you ignore AbortController, you keep spending after the UI has moved on. If you retry forever, a rate limit becomes a thundering herd.

Key takeaways

  • Retry connection failures, 429, and 5xx. Do not retry 400, 401, 403, or a 429 that means the spend cap is gone.
  • One retry budget for the whole stack. SDK retries plus your retries plus a gateway multiply load.
  • Timeouts are three numbers: connect, time to first token, and a hard deadline for the user request.
  • Fall back to a smaller model after the budget, and label the answer as degraded.
  • Tools need idempotency keys. Generation retries never wrap a write that may have succeeded.

Two different verbs: generate vs execute

Treat the provider call as read-ish: same messages in, sampled tokens out. A duplicate completion wastes money and can confuse logs. It rarely double-charges a customer.

Treat a tool as write-ish until you prove otherwise. search_docs is safe to retry. refund_order is not, unless the payment API honors an idempotency key and you pass the same key on every attempt.

Your loop should look like this:

  1. Call the model with a deadline and an abort signal.
  2. If the model proposes a tool, validate args in your code.
  3. Execute the tool once per logical action, outside the generation retry wrapper.
  4. Feed the tool result back. If the next generation 503s, retry generation, not the tool.

See function calling and tools for the authorization half. This page is the reliability half.

What is actually retryable

Signal Retry? Notes
Network reset, DNS, connect timeout Yes The provider likely never saw the body.
HTTP 408, 409, 429 with Retry-After Yes Sleep at least that long, plus jitter.
HTTP 500, 502, 503, 504 Yes Cap attempts. These cluster.
HTTP 400, 413, context overflow No The next try is the same mistake.
HTTP 401, 403 No Fix auth. Retrying hides the outage.
HTTP 429 from a quota/spend cap No Backoff will not mint more quota.
User abort / AbortError No Stop is a product action, not a flake.
Tool HTTP 200 then your client timed out No Assume it committed. Reconcile, do not replay.

Providers and SDKs already retry some of this (often a couple of times with a short backoff). If you add another loop, set the SDK retries to zero or you will 3 × 3 your traffic during an incident.

A retry policy you can ship

Exponential backoff without jitter synchronizes every crashed client onto the same second. Add jitter. Bound the total wait, not only the attempt count.

const RETRY_STATUSES = new Set([408, 409, 429, 500, 502, 503, 504]);

function backoffMs(attempt: number, retryAfterMs?: number) {
  if (retryAfterMs && retryAfterMs > 0) return Math.min(retryAfterMs, 8_000);
  return Math.min(400 * 2 ** attempt, 8_000) + Math.floor(Math.random() * 250);
}

export async function completeWithRetries(
  run: (signal: AbortSignal) => Promise<Response>,
  opts: { maxAttempts: number; deadlineMs: number; signal: AbortSignal },
): Promise<Response> {
  const deadline = Date.now() + opts.deadlineMs;
  let lastError: unknown;

  for (let attempt = 0; attempt < opts.maxAttempts; attempt++) {
    if (opts.signal.aborted) throw new DOMException('Aborted', 'AbortError');
    const remain = deadline - Date.now();
    if (remain <= 0) throw new Error('deadline exceeded');

    const perAttempt = AbortSignal.any([
      opts.signal,
      AbortSignal.timeout(Math.min(remain, 20_000)),
    ]);

    try {
      const res = await run(perAttempt);
      if (res.ok || !RETRY_STATUSES.has(res.status) || attempt === opts.maxAttempts - 1) {
        return res;
      }
      const retryAfter = Number(res.headers.get('retry-after'));
      const wait = backoffMs(
        attempt,
        Number.isFinite(retryAfter) ? retryAfter * 1000 : undefined,
      );
      await new Promise((r) => setTimeout(r, wait));
    } catch (err) {
      lastError = err;
      if (opts.signal.aborted) throw err;
      if (attempt === opts.maxAttempts - 1) throw err;
      await new Promise((r) => setTimeout(r, backoffMs(attempt)));
    }
  }

  throw lastError ?? new Error('retries exhausted');
}

Call completeWithRetries around the model HTTP request only. Tool execution lives next door:

export async function runTool(
  name: string,
  args: unknown,
  ctx: { userId: string; idempotencyKey: string },
) {
  if (name === 'charge_card') {
    // Do not pass this through completeWithRetries.
    // If we timed out after the provider accepted the charge, a retry
    // double-bills unless the processor dedupes on this key.
    return payments.charge({
      ...parseCharge(args),
      userId: ctx.userId,
      idempotencyKey: ctx.idempotencyKey,
    });
  }

  if (name === 'search_docs') {
    return searchDocs(parseSearch(args));
  }

  throw new Error(`unknown tool: ${name}`);
}

Mint idempotencyKey from the conversation id plus the tool call id, not from Date.now(). A new key on every retry is the same as no key.

Timeouts are not one number

A 10-minute SDK default is a hang with extra steps. Chat UI users leave much earlier.

Clock Typical job If it fires
Connect / TLS A few seconds Retry. Nothing useful started.
Time to first token UX budget for the blank bubble Cancel, maybe fallback. The user already bounced.
Idle between streamed tokens Detect a stuck stream Abort. Partial text is not a finished answer.
Overall deadline Whole user request, including retries Return an error. Do not start attempt 4 at second 59.

Streaming changes the math. You can have a short first-token timeout and a longer total timeout. A non-streaming JSON extract needs a timeout that covers the whole object, or you JSON.parse a truncated string and invent a repair loop you did not want.

Wire the browser signal through. AbortController is the same primitive you use for search. The route should pass req.signal (or the client abort) into the provider SDK. Swallowing abort and retrying is how you generate answers for tabs that closed.

Fallbacks without surprise answers

After the retry budget, switching to a smaller or other-provider model is reasonable. Doing it in parallel on every request doubles spend. Doing it silently on a high-stakes task hides a capability drop.

Keep fallbacks honest: same output schema, one winner (ignore late cancelled attempts), a degraded flag on the trace, and no fallback for 401s or context-too-long. Those need a working key or a smaller prompt.

Routing tables belong in a gateway. This page only needs: retry, then one fallback, then fail.

Footguns

  • Retrying a stream from byte 0 after tokens already painted. Use a new message id or you duplicate the bubble.
  • Ignoring Retry-After on 429. You become the outage.
  • Per-attempt timeout longer than the remaining deadline. Attempt 3 starts with 1s left and still burns a generation.
  • Frontend retries plus server retries. Surface 429s in the composer (rate limits).
  • Logging the full prompt on every retry. Redact. Keep a hash and a prompt version.

Interview answer (out loud)

“LLM retries are for transport flakes, 429s, and 5xx on the generation call. I use exponential backoff with jitter, honor Retry-After, and cap total wait. I do not retry 4xx except 429, and I never retry a side-effecting tool unless it is idempotent with a stable key. Timeouts are connect, first token, and a deadline. The browser abort signal cancels the in-flight attempt. After the budget I may fall back to a smaller model that still satisfies the same schema, and I mark that answer as degraded.”

Further reading (primary sources)

Related guides