Frontend Beauty
ESC

Type to search the knowledge base.

Tokens and Context Windows

Tokens are the billing unit and the memory limit. How context windows fill up, why counts surprise you, and how to budget a request.

beginner6 min read
  • tokens
  • context-window
  • cost

You pasted a 20k-word PDF into the prompt. The API said the request was about 32k tokens and rejected it, or it ate the budget you needed for the answer. Words are not tokens. A context window is a token budget, not a page count and not “memory” in the product sense.

Tokens are the billing unit and the working-memory limit. Input tokens, output tokens, tool schemas, images, and (on some models) hidden reasoning all share that window. Cached input can cost less than uncached input. It still occupies the window.

If you budget in English words, you will be surprised by code, tables, CJK, and invoice ids. Count with a tokenizer. Pack on purpose.

Key takeaways

  • The context window is input plus the tokens you still want to generate. Reserve output first.
  • System prompt, tools, history, retrieved chunks, and the new user turn all compete for the same budget.
  • Cached tokens still count against the window. Caching changes price and prefill time, not capacity.
  • Word count and string.length are the wrong meters. See tokenizer gotchas.

What a token is, without the research paper

A token is a chunk of text the model’s tokenizer emits: a word, a piece of a word, a run of punctuation, a leading space glued to a word, sometimes a single digit. The model does not read characters. It reads this sequence.

That is why two strings with the same character length can cost different amounts, and why a short identifier can be several tokens. Do not memorize a “4 characters per token” rule as a budget. It is a rough English average, and it lies for the inputs frontend apps actually send (stack traces, JSON, MDX).

The context window is the maximum number of those tokens the model can attend to for one request: everything you send, plus the completion it is about to write. Overflow is not “it will ignore the middle.” Older APIs reject the request. Newer ones may accept and then stop with a window-exceeded reason mid-generation. Either way, you lost the answer.

The packing diagram

Treat one request as a suitcase. If you overfill it, something falls out, usually the thing you needed.

Slice What belongs here Typical overfill
System Role, policy, output contract A style-guide novel
Tools Names, JSON schemas, descriptions Forty tools, every turn
History Prior user, assistant, and tool messages Untrimmed transcript
Retrieved Chunks you chose to stuff in Top-20 “just in case”
User turn The actual question, plus any live state Hidden dumps (full DOM, whole repo)
Generation max_tokens you reserved Counting only the input

All of those rows share one number: the model’s window. Prompt caching does not remove a row. It can make a stable prefix cheaper. See prompt caching.

type WindowBudget = {
  limit: number;
  system: number;
  tools: number;
  history: number;
  retrieved: number;
  userTurn: number;
  maxOutput: number;
};

function remainingForRetrieval(b: WindowBudget): number {
  const used =
    b.system + b.tools + b.history + b.userTurn + b.maxOutput;
  return b.limit - used;
}

// Failure mode: packing retrieved chunks first, then discovering
// maxOutput no longer fits. The API truncates the answer, not the PDF.

remainingForRetrieval going negative means you already lost. Drop history or chunks before you call the model. Do not hope the vendor will “figure it out.”

How to budget a request

Do this in order. Reversing it is how a docs chatbot answers from the footer.

  1. Fix maxOutput. A citation-backed paragraph might need a few hundred tokens. A long agent trace needs more. If you leave this as the model default, a rambling answer can crowd out the next turn.
  2. Count the stable prefix: system + tool schemas. This should change on deploy, not per keystroke.
  3. Count the new user turn (and any tool result you are about to append).
  4. Fill history from the tail. Newest turns first. Oldest turns drop. A rolling summary is optional; an unbounded array is not.
  5. Fill retrieved chunks into whatever is left. Hard-cap the count and the token sum. Rank first, then pack.
type PackedMessage = { role: 'user' | 'assistant' | 'tool'; content: string };

function packHistory(
  turns: PackedMessage[],
  tokenCount: (text: string) => number,
  budget: number,
): PackedMessage[] {
  const kept: PackedMessage[] = [];
  let used = 0;
  for (let i = turns.length - 1; i >= 0; i--) {
    const cost = tokenCount(turns[i].content) + 8; // role overhead, schematic
    if (used + cost > budget) break;
    kept.push(turns[i]);
    used += cost;
  }
  return kept.reverse();
}

Call a real counter (tiktoken, a provider count-tokens endpoint, or your vendor SDK). The + 8 is a stand-in for per-message framing. Framing is real. Message roles and tool envelopes are not free.

On the client, this is the same instinct as HTTP caching: you do not ship the whole origin on every request. You send what the next computation needs.

Counts that surprise people

Images and PDFs. Vision and document inputs tokenize as patches or pages, not as the alt text you had in mind. A “short” screenshot can dwarf the user question.

Tool schemas. A verbose description on every function is prompt text you pay for on every turn, including the ones that never call the tool.

You must resend history. The model does not keep a server-side brain for your chat id unless you use a vendor session feature. Default Chat Completions / Messages APIs are stateless. The window fills because you keep appending.

Output is in the window. usage after the fact tells you what happened. Budgeting requires a count before send, plus a reserved output slice. Anthropic documents this as input phase then output phase sharing one window; OpenAI’s usage fields split prompt vs completion, but the limit is still one context.

Long context is not free attention. A 200k or 1M window still misses the middle and still bills you. Stuffing the monorepo is a product decision, not a flex. Long context windows covers when dumping loses to retrieval.

When the context window is full but the answer is still wrong

Overflow is the loud failure. The quiet one is a full window of the wrong tokens: duplicate chunks, CSS from a scraped page, three-year-old policy, a system prompt that restates the same rule six times.

Cost follows the same shape. You pay for tokens you send, even if the model ignores them. LLM cost optimization is mostly “stop sending filler,” then cache the prefix you cannot drop.

If a request is near the limit, shrink in this order: retrieved extras, old history, tool descriptions the turn cannot use, then system-prompt essays. Never shrink maxOutput to zero to “make it fit.” You will get a stop-reason of length and a UI that looks like a hang.

Out loud, in an interview

“A context window is a token budget for one request: system, tools, history, retrieved text, the user turn, and the completion. Words are the wrong unit, so I count with the model’s tokenizer. I reserve max output first, pack newest history, then fill retrieval into whatever is left. Caching can make a stable prefix cheaper. It does not make the window bigger. If I blow the limit, I drop context in code, I do not hope the model will skip it.”

Further reading (primary sources)

Related guides