A customer says the support agent gave them the wrong return policy. Without a trace, the team can only guess. With one, they open that conversation and see each step: the user’s question, the search query the agent wrote, the three documents it retrieved (one was out of date), the prompt it built from them, the model’s answer, and how long and how many tokens each step took. The bug is in retrieval, not the model.
Each step is a span, and spans nest inside a trace, the same structure used for tracing ordinary distributed software. OpenTelemetry has published GenAI conventions so that model calls, agent steps and tool calls are recorded with common attribute names, which lets traces move between tools. Traces can contain user data and secrets, so redact and restrict them like any log.
In episode 39 at 24:11, Vercel CTO Malte Ubl explains why agents make tracing unavoidable: the business logic is written in words and the model finds its own path, so the trace is the only record of what it actually did, and it isn’t the same each time (24:46). The episode was sponsored by Galileo, which Ubl mentions working with at 44:37.
Collecting traces is the easy part. In episode 37 at 10:01, Hamel Husain suggests reading traces until you stop learning anything new, with about 100 as a starting heuristic, and later describes building a viewer that shows each trace the way the user saw it (21:53). He names Galileo’s trace viewer as one option; episode 37 was sponsored by Galileo, whose CTO co-hosts it (0:35).
Tracing explains how a request ran; evaluation judges whether the result was good. Teams turn interesting traces into eval cases. See AI observability for the wider practice and audit trail for records kept for accountability.
Hear it from the guest
“You need tracing now. Because the trace tells you what the thing did. It's not always the same trace.”
“Keep looking at traces until you're not learning anything new.”
Quotes lightly edited to remove filler words.
Sources
- OpenTelemetry: Traces — Defines a trace as the path of a request through an application, made of spans for individual operations; checked October 24, 2026.
- OpenTelemetry: GenAI semantic conventions repository — Home of OpenTelemetry's semantic conventions for generative AI spans, events, metrics and agents, moved out of the main semantic-conventions repository; checked October 24, 2026.
- MLflow: LLM Tracing and Agent Observability — Describes traces as a view beneath the abstractions of LLM and agent frameworks, used to locate where issues occur and to collect evaluation data; checked October 24, 2026.
