Frontend Beauty
ESC

Type to search the knowledge base.

LLM Cost Optimization

Cost is tokens × price × retries, plus the tools you invoked. Cache, route, shrink context, and stop looping before you negotiate a contract.

intermediate6 min read
  • cost
  • llmops
  • caching

The model is not what blew the bill. The 12-step agent that re-sent a 40-page system prompt on every hop, with no cache and no max steps, is. LLM cost optimization is arithmetic: tokens × price × retries, plus every tool you invoked, plus the judge you ran on the side. Negotiate a contract last. Fix the loop first.

You will not see this in a toy chat box. You will see it the week you add retrieval, tools, and “try again” on timeouts.

Key takeaways

  • Measure input, output, cached input, and tool calls per user-visible turn, not per HTTP call.
  • The first levers are: smaller model for the task, prompt cache on a stable prefix, less context, fewer steps.
  • Output tokens dominate latency and often dominate spend. Cap them. Ask for short answers when the UI is short.
  • Retries and agent loops multiply everything. Bound them. Do not retry successful side effects.
  • Do not quote list prices in dashboards as truth. Use your billed usage fields; they already include cache discounts.

LLM cost optimization is a loop, not a model

One user message can be many generations. Write the bill in code so product people can read it.

type Usage = {
  inputTokens: number;
  cachedInputTokens: number;
  outputTokens: number;
};

type TurnBill = {
  generations: number;
  tools: number;
  usage: Usage;
};

function addUsage(a: Usage, b: Usage): Usage {
  return {
    inputTokens: a.inputTokens + b.inputTokens,
    cachedInputTokens: a.cachedInputTokens + b.cachedInputTokens,
    outputTokens: a.outputTokens + b.outputTokens,
  };
}

type Completion = {
  usage: Usage;
  toolCall?: { name: string; args: unknown };
};

type CompleteFn = (messages: unknown[]) => Promise<Completion>;

const complete: CompleteFn = async (messages) => {
  const res = await fetch('/api/complete', {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ messages }),
  });
  if (!res.ok) throw new Error(`complete ${res.status}`);
  return res.json() as Promise<Completion>;
};

async function runTurn(opts: {
  messages: unknown[];
  maxSteps: number;
  complete?: CompleteFn;
  runTool: (call: { name: string; args: unknown }) => Promise<string>;
}): Promise<TurnBill> {
  let usage: Usage = { inputTokens: 0, cachedInputTokens: 0, outputTokens: 0 };
  let tools = 0;
  let generations = 0;
  const messages = [...opts.messages];
  const generate = opts.complete ?? complete;

  for (let step = 0; step < opts.maxSteps; step++) {
    const out = await generate(messages);
    generations += 1;
    usage = addUsage(usage, out.usage);
    if (!out.toolCall) break;
    tools += 1;
    const observation = await opts.runTool(out.toolCall);
    messages.push({ role: 'tool', content: observation });
  }

  return { generations, tools, usage };
}

Attach this to the trace. p95 tokens per turn is the number that should scare you, not the demo’s single short reply.

Relative prices (no dollars): uncached input costs more than cached input. Output usually costs more per token than input. A larger model costs more than a smaller one for the same tokens. A loop of 8 generations on the large model will beat any “we picked a cheap provider” story.

Levers, in the order you should pull them

  1. Stop unbounded work. maxSteps, maxTokens, timeouts, no retry on 2xx tool results. Disable Send while streaming so users cannot stack in-flight prefills. This is the difference between a feature and a runaway worker.
  2. Right-size the model. Classification, extraction, and routing often do not need the frontier model. See model routing. Keep the large model for the turns your eval says need it.
  3. Cache a stable prefix. System prompt, tool schemas, and the static style guide belong at the front of the prompt so prompt caching can hit. User text and retrieved chunks go last. Cached input is cheaper than uncached input; treat hit rate as a product metric.
  4. Retrieve instead of dump. Stuffing the handbook every turn is a tax. RAG with a token budget is usually cheaper and better. Long context is a cost choice, not a flex.
  5. Shrink outputs. “Be brief” plus a low maxTokens plus structured JSON with short keys. Streaming does not make tokens free.
  6. Semantic cache for FAQs. If the question is close to a prior one and the answer is not user-specific or time-sensitive, return the stored reply. See semantic cache. Never cache a mutating tool result.
  7. Batch offline work. Nightly summarization and eval runs belong on batch APIs when the provider offers them. Interactive chat does not.
function packForCache(args: {
  system: string;
  toolsJson: string;
  history: ChatMessage[];
  user: string;
}): ChatMessage[] {
  // Static prefix first: cache identity depends on it.
  return [
    { role: 'system', content: `${args.system}\n\nTOOLS:\n${args.toolsJson}` },
    ...args.history,
    { role: 'user', content: args.user },
  ];
}

If you shuffle tools or rewrite the system prompt every request, you pay full input again. That is a self-inflicted miss. Know the vendor’s cache identity rules: some APIs cache a single prefix breakpoint, so two system messages can miss even when the bytes look stable. Prefer one static prefix block, then the moving suffix.

Measure before you negotiate

Build a weekly view with four numbers:

Metric Tells you
Cache hit rate (cached / input tokens) Prefix stability
Output tokens per turn (p50 / p95) Verbosity and loops
Generations per turn Agent step budget
Share of turns on the expensive model Routing quality

Then join to quality from the golden set. A cheaper model that fails refunds is not cheaper. It is a support queue.

Do not optimize from a spreadsheet of public list prices. Providers discount cache, batch, and committed use. Read usage on the response you actually got.

What “optimizing” gets wrong

  • Dropping the system prompt to save input tokens while destroying the only policy the model had. Put policy in code; keep the prompt short after that.
  • Retrying the whole agent on a 429 without jitter or a budget. You turn a rate limit into a thundering herd. See retries in production.
  • Logging every prompt to an observability vendor billed per GB, then wondering where spend went. Observability is a second bill.
  • Fine-tuning to save tokens before you have an eval. Distillation can shrink models; it can also freeze last month’s bugs into weights.

Frontend-specific: disable Send while streaming so users cannot stack in-flight requests. Each extra click is another full prefill.

Interview answer (out loud)

“LLM cost is tokens times price times how many times I called the model, plus tools. I cap steps and output tokens, route easy tasks to a small model, put a stable prefix in front for prompt cache, and retrieve instead of dumping the corpus. I track cache hit rate and p95 tokens per turn on traces. I do not start with vendor negotiations, and I do not retry side effects.”

Further reading (primary sources)

Related guides