LLM Observability: Tracing Agents with OpenTelemetry

LLM observability is the practice of emitting a structured trace for every agent request — one span per step, each tagged with tokens, cost, latency, model, and outcome — so a slow or wrong answer becomes a thing you can read, not a thing you guess at. A multi-step agent that only writes log lines is a black box; the same agent instrumented with OpenTelemetry is a waterfall you can open and point at the span that broke.

Logs tell you it happened; spans tell you the shape

An agent call fans out: embed the query, retrieve, rerank, generate, verify, maybe loop. When the p95 latency creeps up or the bill doubles, scattered log lines can't tell you which step did it — they have no parent, no duration, no shared trace id. A span does. Wrap each step in a span and one request becomes a tree: total time at the root, a labelled child per stage, and attributes hanging off each one. The regression stops being a hunch.

The GenAI semantic conventions make traces comparable

OpenTelemetry ships a standard vocabulary for model calls — gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens. Using the conventions instead of ad-hoc field names means any backend — Grafana, Honeycomb, Langfuse — renders your traces the same way, and cost-per-trace becomes a query over output_tokens rather than a spreadsheet. Standard attributes are what turn one team's telemetry into a dashboard anyone can read.

// wrap the model call in a span tagged with the GenAI conventions
return tracer.startActiveSpan("model.generate", async (span) => {
  span.setAttributes({
    "gen_ai.system": "anthropic",
    "gen_ai.request.model": model,
  });
  const res = await llm.generate(prompt);
  span.setAttributes({
    "gen_ai.usage.input_tokens": res.usage.input,
    "gen_ai.usage.output_tokens": res.usage.output,
    "gen_ai.cost.usd": cost(res.usage),  // derived, for cost-per-trace
  });
  span.end();
  return res;
});

Online tracing is not offline evals — you need both

Observability watches production as it happens: latency, cost, error rates, the actual distribution of traffic. Evals score quality against a fixed set before you ship. They answer different questions — "is it fast and cheap right now?" versus "is it correct?" — and neither covers the other. A trace can tell you a span took four seconds; only an eval tells you the answer it produced was wrong. Wire up tracing for the live system, keep evals for the quality bar.

You can't optimize what you can't see. A trace per request turns "the agent feels slow lately" into "the rerank span doubled on Tuesday" — and that's the difference between debugging and guessing.

Tracing is what makes the latency-first target measurable and autonomous loops auditable. Continue on the roadmap.