When Logs Lie: Tracing LLM Agents with OpenTelemetry

The live RAG concierge on this site ships a structured trace with every request — one span per step, tagged with tokens, cost, latency, model, and outcome — because the first time an agent felt slow I had only scattered logs and no way to prove which step did it. A multi‑step agent that only logs is a black box; wire it with OpenTelemetry and you get a waterfall you can scan and point at the span that blew the budget. In the pattern I call Trinity Architecture, tracing lives in the orchestration layer, not the UI or the model client.

Logs tell you it happened; spans tell you the shape

An agent call fans out: embed the query, retrieve, rerank, generate, verify, maybe loop. When the p95 latency creeps up or the bill doubles, scattered log lines can't tell you which step did it — they have no parent, no duration, no shared trace id. Spans do. Wrap each step and one request becomes a tree: total time at the root, a labelled child per stage, and attributes on each. On IntegrateX we caught a rerank burst exactly this way when canvas executions started queuing; the hunch became one slow span you could fix.

The GenAI semantic conventions make traces comparable

OpenTelemetry ships a standard vocabulary for model calls — gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens. Using the conventions instead of ad-hoc field names means any backend — Grafana, Honeycomb, Langfuse — renders your traces the same way, and cost-per-trace becomes a query over output_tokens rather than a spreadsheet. I route these attributes through a thin Serialization Adapter so telemetry stays lean and comparable without leaking UI or DB shapes across layers.

// wrap the model call in a span tagged with the GenAI conventions
return tracer.startActiveSpan("model.generate", async (span) => {
  span.setAttributes({
    "gen_ai.system": "anthropic",
    "gen_ai.request.model": model,
  });
  const res = await llm.generate(prompt);
  span.setAttributes({
    "gen_ai.usage.input_tokens": res.usage.input,
    "gen_ai.usage.output_tokens": res.usage.output,
    "gen_ai.cost.usd": cost(res.usage),  // derived, for cost-per-trace
  });
  span.end();
  return res;
});

Online tracing is not offline evals — you need both

Observability watches production as it happens: latency, cost, error rates, the actual distribution of traffic. Evals score quality against a fixed set before you ship. They answer different questions — "is it fast and cheap right now?" versus "is it correct?" — and neither covers the other. A trace can tell you a span took four seconds; only an eval tells you the answer it produced was wrong. On streamerOS we even sampled traces to avoid backpressure and keep 60fps renders while offline evals guarded quality.

You can't tune what you can't see. A trace per request turns "the agent feels slow lately" into "the rerank span doubled on Tuesday" — and that's the difference between debugging and guessing.

Tracing is what makes the latency-first target measurable and autonomous loops auditable — it is why the concierge on this page can show live execution traces instead of asking you to trust it. The judgment I bring to a team is that: an agent you cannot see is an agent you cannot operate, so the trace is part of the design, not an afterthought. Continue on the roadmap.