Frontend Beauty
ESC

Type to search the knowledge base.

Tracing and Observability for LLM Apps

A trace is every prompt, tool call, token count, and latency span for one user request. Without it you cannot debug, eval, or explain a bill.

intermediate5 min read
  • observability
  • tracing
  • llmops

“The bot was weird” is not a ticket. A trace id is. LLM observability means you can reconstruct one user request: the prompt version, retrieved chunk ids, every tool call, token counts, and where the time went. Without that tree you cannot debug, you cannot sample for evals, and you cannot explain the bill.

This is OpenTelemetry thinking applied to a sampler. Spans nest. Attributes are cheap. The payload is the prompt, so redaction is part of the instrumentation, not a later cleanup job.

Key takeaways

  • One user request is a root span with children for retrieval, each generation, and each tool.
  • Store prompt version, model id, token counts, cache hits, and tool names on the spans. That is how you debug cost and quality.
  • Redact PII before the exporter. Logs that contain the user’s passport number are an incident, not a feature.
  • Traces are the raw material for golden rows and online judges. If you did not record it, you cannot grade it.
  • A conversation id is not a trace id. Users retry. Agents loop. You need both.

LLM observability is a tree of spans

A chat turn is not one HTTP call in your process. The browser hits your route handler. You retrieve. You generate. The model asks for a tool. You run it. You generate again. Each of those is a span under the same trace.

type Span = {
  traceId: string;
  spanId: string;
  parentSpanId?: string;
  name: string;
  startMs: number;
  endMs?: number;
  attrs: Record<string, string | number | boolean>;
  status: 'ok' | 'error';
};

async function withSpan<T>(
  parent: Span | undefined,
  name: string,
  attrs: Span['attrs'],
  fn: (span: Span) => Promise<T>,
): Promise<T> {
  const span: Span = {
    traceId: parent?.traceId ?? crypto.randomUUID(),
    spanId: crypto.randomUUID(),
    parentSpanId: parent?.spanId,
    name,
    startMs: Date.now(),
    attrs,
    status: 'ok',
  };
  try {
    const result = await fn(span);
    return result;
  } catch (err) {
    span.status = 'error';
    span.attrs['error.type'] = err instanceof Error ? err.name : 'unknown';
    throw err;
  } finally {
    span.endMs = Date.now();
    await exporter.send(span);
  }
}

Name spans after the work, not after the library: rag.retrieve, llm.generate, tool.get_order. When an agent loops, you get a comb of llm.generate / tool.* children. That comb is the agent failure report.

Fields that earn their keep on a generation span

Put these on every llm.generate. If a field is missing, you will invent it during an incident.

Attribute Why
user.id (hashed) Tenant isolation, abuse
conversation.id / turn Thread vs this request
prompt.version Which template shipped
model.id Pin, not “latest”
gen.input_tokens / gen.output_tokens Cost and context pressure
gen.cached_input_tokens Cache hit rate
gen.ttft_ms User-felt latency
retrieve.chunk_ids Grounding debug
tool.names What the model asked for

You do not need to store the full rendered prompt on every production turn if you cannot redact it. You do need a version id plus the variables (order id, locale) so you can re-render offline. Sample a percentage of turns with full payloads for evals.

async function generateTraced(span: Span, req: CompletionRequest, promptVersion: string) {
  const t0 = Date.now();
  let ttftMs: number | undefined;
  const stream = await provider.stream(req, {
    onFirstToken: () => {
      ttftMs = Date.now() - t0;
    },
  });
  const out = await readAll(stream);

  span.attrs['prompt.version'] = promptVersion;
  span.attrs['model.id'] = req.model;
  span.attrs['gen.input_tokens'] = out.usage.inputTokens;
  span.attrs['gen.output_tokens'] = out.usage.outputTokens;
  span.attrs['gen.cached_input_tokens'] = out.usage.cachedInputTokens ?? 0;
  if (ttftMs !== undefined) span.attrs['gen.ttft_ms'] = ttftMs;
  return out;
}

Token and cache fields are how cost stops being a monthly surprise.

Redact before you store

The span exporter is a new data store. Treat it like logs that happen to include user content.

  1. Strip Authorization headers and provider API keys. They should never have been near the span in the first place.
  2. Run the same PII rules you use for support transcripts: emails, card-shaped numbers, government ids.
  3. Cap payload size. A 200k-token dump will blow your observability bill and still be unreadable.
  4. Separate “debug full prompt” (restricted access, short TTL) from “metrics attributes” (long retention).

If you cannot redact, do not attach the raw user message. Attach a hash and the prompt version. You can still join to your own DB under access control.

Traces become evals

The useful loop:

  1. Production trace fails (user flag, tool error, empty retrieval, judge sampled online).
  2. Engineer labels the turn: wrong intent, bad chunk, bad tool arg.
  3. Row lands in the golden dataset.
  4. CI gate replays the same prompt version.

None of that works if the trace dropped the chunk ids or the prompt version. Observability is not a dashboard of p50 latency. It is the ability to replay.

Frontend notes that people skip: send the trace id back on the SSE stream so the UI’s “report this” button attaches it. Users mash Stop: honor AbortController and mark the span status=error with attrs['gen.aborted']=true so cancelled turns do not look like model outages.

Footguns

  • Logging only the last assistant string. You will not see the tool loop that caused it.
  • A new trace per tool call. You lose the parent. Keep one trace per user turn.
  • Unpinned model: gpt-x in spans. Aliases move. Store the resolved snapshot id the API returned when you have it.
  • PII in span names. llm.generate is fine. llm.generate.user@company.com is not.
  • No sampling policy. 100% full-payload traces will become the most expensive part of the app. Sample payloads; keep 100% of counters.

Interview answer (out loud)

“LLM observability is a trace per user turn: nested spans for retrieval, generations, and tools, with prompt version, model id, tokens, cache hits, and TTFT on the generation span. I redact PII before export and send the trace id to the UI so ‘it was weird’ becomes a replay. Those traces are how I grow the golden set. Without them I cannot debug quality or the bill.”

Further reading (primary sources)

Related guides