Frontend Beauty
ESC

Type to search the knowledge base.

Long Context Windows

Million-token windows still miss the middle and still cost money. When to stuff the repo in, and when to retrieve instead of dumping.

intermediate5 min read
  • context
  • long-context
  • cost

You pasted the monorepo. The model answered from README.md and ignored the module that implements the bug. A long context window raised the ceiling. It did not make the haystack searchable. Million-token SKUs still miss the middle, still bill for every input token (cached or not), and still compete with the tokens you need for the answer.

Stuffing is a valid tactic when you already chose the N documents. It is a bad default for “the whole repo, just in case.” Retrieval and file tools exist because attention is uneven and cost is linear in what you send.

Key takeaways

  • A long context window is a larger working memory, not a guarantee the model read page 400.
  • Position effects are a product risk. Needle-in-haystack demos are not your eval.
  • Dump when the user (or your packer) already selected a small, relevant set.
  • Retrieve or read_file when only a few paths matter.
  • Cache a stable prefix. Do not reshuffle the haystack every turn.

A long context window is still a window

Providers now ship windows from tens of thousands of tokens up through the million-token class (Gemini’s long-context docs describe 1M-token models; Anthropic documents 200k as common, with longer windows on some models). The number on the marketing page is the sum of what you send and what you generate, plus tools and thinking tokens on stacks that count them. Overflow is an API error or a stop reason. Silence in the middle of the prompt is worse: a fluent answer that never saw the file.

Do not quote a needle-in-haystack percentage. Those numbers are benchmark- and model-specific, they move, and they are not your corpus. Treat “can it find a random sentence we buried?” as a risk to test on your traces, not a spec you copy from a blog.

type Source = { id: string; path: string; text: string };

export function shouldDump(opts: {
  sources: Source[];
  count: (s: string) => number;
  window: number;
  reserved: number;
  query: string;
}): 'dump' | 'retrieve' {
  const total = opts.sources.reduce((n, s) => n + opts.count(s.text), 0);
  const fits = total + opts.count(opts.query) + opts.reserved < opts.window * 0.5;
  const selected = opts.sources.length <= 12;
  const sameJob = opts.sources.every((s) => s.path.startsWith('docs/refunds/'));
  if (fits && selected && sameJob) return 'dump';
  return 'retrieve';
}

Half the window is a conservative packer, not a law. Leave room for the question, the system prompt, and the answer. If you are already at 80% before the user speaks, you are going to drop the middle whether the limit is 128k or 1M.

Position, noise, and README gravity

Three effects show up in real apps without needing a fake score:

  1. Edges beat the middle. Put long documents first, the question last (Anthropic’s long-context guidance). Repeat a one-line hint next to the query if a fact must survive.
  2. Noise wins. Nav, license files, generated locks, and dist/ look like English. They are not the bug. Strip chrome before you stuff. The RAG on PDFs and HTML problem is the same here: layout garbage retrieves as if it were policy.
  3. Short, central files get cited. README.md is short, high in the tree, and full of confident sentences. The 2k-line reducer is none of those. If you dump a repo, you should expect README-shaped answers unless you pin the path.

Context engineering is how you fight this: pin, retrieve, summarize, drop. A long window just gives you more rope.

Dump, retrieve, or cache?

Tactic Use when Breaks when
Stuff selected docs Compare these 6 PDFs; audit this one module You send the monorepo “for awareness”
Retrieve / file tool Unknown path, large tree, per-question subset The task is “diff these two known files” and you still search
Prompt cache Stable prefix: system, tool schemas, a handbook You reorder or rewrite the prefix every turn

Cached input is cheaper than uncached input. It is not free, and it still occupies the window. Anthropic is explicit: caching changes what you pay, not whether the tokens count. If you reshuffle files to “help” the model, you miss the cache and you scramble positions.

RAG is still the right default for “ask the docs.” Long context is the right default for “the user attached three contracts, quote them against each other.” Mixing them is normal: retrieve 20, dump the top 5, cache the system prompt.

Cost scales with tokens × retries × agent steps. A million-token dump inside a 12-step loop is how a “cheap” model becomes the expensive one. Cap what you send per step. Prefer a file tool that reads 400 lines over a prefix that contains 400 files.

What to test instead of a leaderboard

On your own golden questions:

  • Move the supporting paragraph to the start, middle, and end of the packed prompt. Does the answer stay entailed?
  • Add a distractor README. Does the citation still point at the module?
  • Measure input tokens and cache hit rate, not only “quality.”
  • Refuse when retrieval is empty and you chose not to dump. A long window is not an excuse to guess.

If the answer cites a path you never sent, that is a hallucination, not a context-size win. Verify ids in code.

Interview answer (out loud)

“A long context window raises the token ceiling. It does not make the model a search engine. I stuff when I already selected a small relevant set, retrieve or read files when the tree is large, and cache a stable prefix. I would not quote needle-in-haystack scores. I would test position and distractors on my own traces, and I would leave room for the answer. README-only replies after a repo dump are a packing failure.”

Further reading (primary sources)

Related guides