Frontend Beauty
ESC

Type to search the knowledge base.

Batch vs Realtime Inference

Chat is realtime. Embeddings, evals, and night-time summaries can wait. Use batch inference when the user is not in the loop.

intermediate5 min read
  • inference
  • batch
  • cost

Re-embedding 80k chunks through the chat API is how you discover batch inference exists: on the invoice, while a progress spinner lies to nobody. Chat is realtime. Embeddings, eval sweeps, and nightly summaries can wait. If no human is staring at a cursor, you should not be on the interactive quota.

Realtime is a streaming HTTP request with a user in the loop. Batch is a job: submit, poll, load results, reconcile by id. Mixing them is a cost bug and a latency bug wearing the same clothes.

Key takeaways

  • If the UI needs tokens now, use the realtime API and stream. If it does not, queue a batch job.
  • Batch is cheaper and has a separate rate-limit pool. It is also slower and not for side-effecting tools.
  • Jobs must be idempotent. A content hash as custom_id beats “run it again and hope.”
  • The UX for batch is poll, webhook, or email, never a 20-minute chat spinner.
  • Do not send the same 80k embedding calls through your chat route “because the SDK was already imported.”

When batch inference is the right API

Ask one question: is a person blocked on this token?

Work User in the loop? Default
Chat reply, copilot inline, support composer Yes Realtime, stream, abort
Re-embed a corpus after a parser change No Batch embeddings
Nightly summarize closed tickets No Batch chat completions
Sweep a 500-case golden set in CI No (CI is the user) Batch, or a worker with its own pool
Classify an upload so the next screen can render Sort of Realtime if the next click needs it; job + poll if it can wait
Agent tool loop (refund_order) Yes, and stateful Realtime only. Batch cannot hold your loop.

Providers document batch as async completions with a completion window measured in hours (OpenAI’s Batch API is a 24h window; Anthropic’s Message Batches API is the same idea). They bill it cheaper than the interactive endpoint and give it headroom so you do not 429 your chat users. Read the live guide before you pin a discount in a slide: OpenAI Batch, Anthropic batch processing.

Realtime still owns:

  • Time to first token. Users judge the blank box. TTFT.
  • Abort. A batch job does not care that the tab closed.
  • Multi-turn tool loops. A batch line is one request, not an agent.

Submit, poll, reconcile

A batch is a file (or a list) of independent requests. You need a custom_id you minted, because results can come back out of order.

type EmbedJob = {
  custom_id: string;
  method: 'POST';
  url: '/v1/embeddings';
  body: { model: string; input: string };
};

function embedJobs(chunks: { id: string; text: string; hash: string }[]): EmbedJob[] {
  return chunks.map((c) => ({
    custom_id: `embed:${c.id}:${c.hash}`,
    method: 'POST',
    url: '/v1/embeddings',
    body: { model: 'text-embedding-3-small', input: c.text },
  }));
}

function toJsonl(jobs: EmbedJob[]): string {
  return jobs.map((j) => JSON.stringify(j)).join('\n');
}

type BatchStatus = 'validating' | 'in_progress' | 'completed' | 'failed' | 'expired';

async function waitForBatch(
  getStatus: () => Promise<{ status: BatchStatus }>,
  opts: { intervalMs: number; deadlineMs: number },
) {
  const start = Date.now();
  for (;;) {
    const { status } = await getStatus();
    if (status === 'completed') return;
    if (status === 'failed' || status === 'expired') {
      throw new Error(`batch_${status}`);
    }
    if (Date.now() - start > opts.deadlineMs) throw new Error('batch_timeout');
    await new Promise((r) => setTimeout(r, opts.intervalMs));
  }
}

embed:${id}:${hash} means a rerun after a no-op parser change submits nothing new, or overwrites the same vector. That is the ingestion habit from document pipelines: idempotent upserts, not “launch 80k chat completions and go home.”

Partial failure is normal. A batch can finish with a mix of succeeded and errored lines. Your job code should apply the successes, retry the failures as a new batch, and never treat “batch completed” as “every row is good.”

UX when the model will not speak for hours

Do not reuse the chat composer.

  1. Accept the job. Return a jobId immediately. Persist pending on the document / report row.
  2. Show a non-blocking state. “Weekly summary queued. We will mail you, or check back on this page.”
  3. Poll or push. Browser: poll every few seconds with backoff, or subscribe if you already have a realtime channel. Server: webhook from your worker, not from the model vendor if you can avoid it.
  4. Make results bookmarkable. The summary is a resource, not a chat turn that vanishes.
  5. Never hide this behind Send. If you must offer “summarize this thread,” say it is a job.

Frontend caching still applies: do not re-fetch a completed report on every focus. See frontend caching strategies. Chat stays a live stream. Different product.

Footguns

  • Batching a side-effecting tool. There is no human in that loop and no good story for half-applied refunds. Realtime, approval, idempotency. Retries.
  • Using batch for the user’s first token. You will lose to a spinner. If you need cheap, use a smaller realtime model, cache the prefix, or shrink context. Cost.
  • One giant JSONL with no chunking of the job itself. Providers cap requests per batch and bytes per file. Split on your side.
  • Forgetting vendor retention. Batch inputs are files. They have a lifetime. Do not put raw PII in a JSONL you cannot TTL. PII.
  • Polling from 40 browser tabs. The browser polls your job endpoint, with session auth, not the provider.

CI evals are a gray area. A 30-case gate can be realtime and still be fine. A 5,000-case nightly sweep is a batch (or a worker on the batch endpoint). Do not put the sweep on the same key your production chat uses without thinking about 429s.

Interview answer (out loud)

“Chat is realtime: stream, abort, TTFT. Embeddings, eval sweeps, and nightly summaries are batch jobs with a custom id, a poll loop, and idempotent writes. Batch is cheaper and slower, and it is the wrong place for a tool loop. I would never re-embed a corpus through the chat completions route.”

Further reading (primary sources)

Related guides