Frontend Beauty
ESC

Type to search the knowledge base.

Context Engineering

Context engineering is selecting, compressing, and ordering what the model sees. Prompts are one input. History, tools, and retrieval are the rest.

intermediate5 min read
  • context
  • prompts
  • agents

A 200k window stuffed with noise is a smaller working set than 8k of the right files. Context engineering is the job of selecting, compressing, and ordering every token the model will attend to: system prompt, tools, retrieved chunks, history, and the last tool dump. Prompt wording is one input. Most production failures are packing failures.

If you already budget a bundle, this will feel familiar. The context window is a payload with a hard Content-Length. Overflow is a 400. Underflow of the useful bits is a fluent wrong answer.

Key takeaways

  • Context engineering is packing: budget, pin, retrieve, summarize, drop.
  • Order matters. Models often use the start and end of a long prompt more reliably than the middle.
  • Tool results are first-class context. A 30k JSON dump will crowd out the user’s question.
  • Pin the few facts that must not drift (policy, ids, schema). Retrieve the rest.
  • A smaller true window beats a stuffed one. Cached prefixes only help if they stay stable.

Context engineering is a packing problem

Count tokens with a tokenizer, not with string.length. See tokens and context windows. Leave room for the model’s reply. Everything else is competing for the same array.

Slot What belongs Default instinct
System Role, hard policy, output contract Keep short. Not a junk drawer.
Tools Schemas the model may call Few, tight enums.
Pinned Tenant, user id, current ticket, non-negotiable quotes Always present, stable prefix.
Retrieved Chunks for this question Top-k after filters, not the corpus.
History Recent turns, or a summary plus last N Summarize the middle.
Observation Last tool results Truncate, schema-slim, or file-offset.
Generation The answer max_tokens is part of the budget.

Anthropic’s long-context tips are blunt: put the long documents first, put the query last, and ask for quotes before the answer. That is packing, not poetry.

type PackInput = {
  system: string;
  toolsTokens: number;
  pinned: string;
  retrieved: Array<{ id: string; text: string }>;
  history: string;
  observation: string;
  query: string;
  window: number;
  maxOut: number;
  count: (text: string) => number;
};

export function packPrompt(p: PackInput): { messages: Array<{ role: 'system' | 'user'; content: string }>; dropped: string[] } {
  const dropped: string[] = [];
  const reserve = p.maxOut + p.toolsTokens;
  let used = p.count(p.system) + p.count(p.pinned) + p.count(p.query) + reserve;

  const chunks: string[] = [];
  for (const c of p.retrieved) {
    const n = p.count(c.text);
    if (used + n > p.window) {
      dropped.push(c.id);
      continue;
    }
    chunks.push(`<doc id="${c.id}">${c.text}</doc>`);
    used += n;
  }

  let history = p.history;
  if (used + p.count(history) > p.window) {
    history = history.slice(-2_000);
    dropped.push('history_truncated');
  }

  let observation = p.observation;
  if (used + p.count(observation) > p.window) {
    observation = observation.slice(0, 1_500) + '\n[truncated]';
    dropped.push('observation_truncated');
  }

  const user = [
    p.pinned,
    chunks.join('\n'),
    history,
    observation ? `<tool_result>${observation}</tool_result>` : '',
    p.query,
  ]
    .filter(Boolean)
    .join('\n\n');

  return {
    messages: [
      { role: 'system', content: p.system },
      { role: 'user', content: user },
    ],
    dropped,
  };
}

Log dropped. If the answer cites a chunk you dropped, that is a packer bug, not a model bug.

Pin, retrieve, summarize, drop

Four verbs. Use them in that order of violence.

Pin the handful of strings that must be in every turn: tenant, locale, “never refund without tool X”, the JSON schema. Pinning is how you keep a prompt cache hit: stable prefix, bouncing user text after the breakpoint.

Retrieve what changes per question. That is RAG or a file-read tool. Do not pre-stuff the handbook.

Summarize old history and fat tool results. Agent memory is three systems (scratchpad, session summary, long-term store). Do not paste all three.

Drop anything you will not grade. Debug logs, previous failed plans, CSS of a retrieved HTML page. Drop is a feature.

Tool results eat the window

Agents die from observations, not from system prompts. A search tool that returns 50 hits at 2k each will shove the user’s question into the cheap seats. Slim the schema: {id, title, score, snippet} not the full document. Offer read(id, offset) for the one hit that matters.

The same rule applies to stack traces and test output. Tail them. The model needs the failing assertion, not the 4,000 lines of coverage.

Lost in the middle is a packing bug

Long-context research (the “lost in the middle” line of work) is the product warning: models often attend more reliably to the beginning and the end of a long prompt than to the buried middle. Do not invent a benchmark score for your app. Do test it. Put the question last. Put must-keep policy first. If a fact has to survive, repeat a short form of it next to the query, or require a quote-then-answer step.

A 200k dump of a repo will still answer from README.md if that file sits in a lucky position and the real module sits in the haystack. That failure is why long context windows are not a replacement for retrieval.

Interview answer (out loud)

“Context engineering is selecting, compressing, and ordering the tokens the model sees: system, tools, pinned facts, retrieval, history, and tool results. I budget the window, pin a stable prefix, retrieve per question, summarize history, and truncate observations. Order matters; the middle of a fat prompt is easy to miss. A 200k pile of noise is worse than 8k of the right files.”

Further reading (primary sources)

Related guides