Frontend Beauty
ESC

Type to search the knowledge base.

Temperature, Top-p, and Sampling

Temperature and top-p change how a model samples the next token. When to pin them down for tools and JSON, and when variation is the feature.

beginner6 min read
  • sampling
  • temperature
  • decoding

Your extractor invented a taxCode field at temperature 0.8. The prompt did not ask for it. The schema did not include it. The bug was the sampler, not a missing adjective in the system prompt.

LLM temperature and top-p do not make a model “more creative” in the marketing sense. They reshape the next-token distribution before a draw. High temperature fattens the tail. Low top-p cuts it. For tools and JSON, you usually want the mode, not the tail. For a brainstorm UI, the tail is the product.

Pin the knobs per route. Do not copy temperature: 0.7 from a tutorial into an invoice parser.

Key takeaways

  • Sampling is a draw from a distribution over the vocabulary. Temperature and top-p change that distribution.
  • Extraction, tool calls, and classifiers want a low temperature. Ideation wants room in the tail.
  • Temperature 0 is not a deterministic lock across providers, hardware, or days.
  • Running both knobs hard at once can starve the distribution. Pick a primary knob.

Next token is a draw, not a lookup

After the prompt, the model produces a logit for every token in the vocabulary. Softmax turns those into probabilities. Decoding then picks one token:

  1. Optionally scale logits by temperature.
  2. Optionally keep only the smallest set of tokens whose probabilities sum to top-p (nucleus sampling).
  3. Draw (or take argmax, depending on the stack).
  4. Append the token, repeat.

That loop is why the same prompt can yield two legal JSON objects, or one legal object and one hallucinated key. Hallucination debugging starts here when the prompt and retrieval already look fine.

type Sampler = {
  temperature: number;
  topP: number;
};

const EXTRACT: Sampler = { temperature: 0, topP: 1 };
const DRAFT_COPY: Sampler = { temperature: 0.8, topP: 0.9 };

async function complete(params: {
  messages: { role: 'system' | 'user'; content: string }[];
  sampler: Sampler;
  maxTokens: number;
}) {
  const res = await fetch('/api/complete', {
    method: 'POST',
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({
      messages: params.messages,
      temperature: params.sampler.temperature,
      top_p: params.sampler.topP,
      max_tokens: params.maxTokens,
    }),
  });
  if (!res.ok) throw new Error(`complete ${res.status}`);
  return res.json() as Promise<{ text: string }>;
}

// Failure mode: DRAFT_COPY on an extractor. Extra keys, flipped enums,
// "creative" dates. The prompt looks innocent in the trace.

Keep sampler settings next to the route, not in a global SDK default. A shared client with temperature: 0.7 will silently tax every JSON call.

LLM temperature vs top-p

Temperature divides the logits before softmax. Values near 0 sharpen the distribution toward the most likely tokens. Higher values flatten it. Provider scales differ: some APIs clip at 1, some at 2. A number is not portable across vendors.

Top-p keeps the nucleus: the smallest set of tokens whose cumulative probability reaches p, then samples inside that set. top_p: 1 means “do not cut.” Lowering it removes the long tail even if temperature is moderate.

They interact. High temperature plus low top-p is not “maximum creativity.” It is a flattened distribution that you then slice. Low temperature plus low top-p can leave one or two tokens and look like a stuck record.

Job Temperature Top-p Why
JSON extract, classifiers 0 to 0.2 1 (or leave default) You want the mode
Tool-call args 0 to 0.2 1 Wrong enum is a prod bug
Support reply with a script 0.2 to 0.4 1 or 0.9 Some phrasing variety, tight facts
Headline / copy ideas 0.7 to 1.0 0.9 to 1 The tail is the feature
Code from a spec low 1 Syntax is not a place to explore

When you need a typed object, prefer structured outputs over “please return JSON” plus temperature folklore. Constrained decoding removes invalid tokens from the support. Sampling still fills the valid fields, so garbage values remain your problem.

Function calling is the same story: the name and argument JSON should be sampled cold. The prose around a tool call, if any, is a separate concern.

Should you touch both knobs?

Usually no. Pick temperature as the product knob. Leave top-p at the provider default unless you have an eval that says otherwise.

Change one variable at a time against a golden set. If an extractor still invents keys at temperature 0, the schema is too loose or retrieval is empty. The sampler is not the next place to look.

Reasoning-style models may ignore or restrict temperature. Read the model card. Putting temperature: 0.8 on a model that does not sample that way is dead config, and it will confuse the next person who reads the route.

Temperature 0 is not a lock

People hear “greedy decoding” and ship snapshot tests against the completion string. Then CI flakes.

Causes that survive temperature: 0:

  • Floating-point and batching. Argmax on nearly tied logits is not a mathematical identity across GPUs.
  • Provider-side changes. A silent model snapshot, a different region, a new decoding kernel.
  • Seeds are best-effort. Some APIs expose seed. Treat it as a debugging aid, not a consensus protocol.
  • Mixture-of-experts and sparse routing. Tiny numeric noise can pick a different expert.

If you need a stable machine result, do not ask the model. Hash, SQL, or a function. Use the model to propose, then verify.

For tests, assert on structured fields (status === 'open'), allowlists, or an eval score. Do not expect(text).toEqual(fixture) on prose.

What still goes wrong

One global sampler. A chatbot SDK default leaks into the billing extractor. Split clients or pass explicit params every call.

“Creative” as a product requirement for facts. If the UI is a form-filler, variation is a defect. Put temperature in the eval report so a prompt PR cannot raise it quietly.

Top-k vs top-p confusion. Some stacks expose top_k (keep k tokens). Do not mix three truncation knobs because a blog post listed them.

Stop sequences. Sampling is not the only control. A stop string still truncates. A tiny max_tokens still cuts JSON in half, and no temperature will repair it.

UI copy that promises “the same answer every time.” You cannot honestly promise that for a sampled API. Promise a schema, a retry budget, and a human path for irreversible actions.

This is closer to flaky network UI than to a pure function. If you already think about flaky tests, treat temperature as a flake amplifier you chose.

Out loud, in an interview

“Temperature and top-p reshape the next-token distribution. I pin temperature near zero for extraction and tool arguments, and I leave headroom only when variety is the feature. I do not crank both knobs at once. Temperature 0 is not deterministic across providers, so I would not snapshot prose. If I need a stable object, I use structured outputs and then validate business rules in code.”

Further reading (primary sources)

Related guides