What is AI observability, and why do you need it in production?

In detail

Why traditional monitoring isn’t enough

Classic monitoring tells you a request returned in 200 milliseconds with a 200 status. For an AI system that says almost nothing, because the response was successful and wrong. The interesting failures — a bad retrieval, a misused tool, an answer that drifted off the question — all happen inside a request that looks fine from the outside.

What observability captures

Observability opens up the request. It traces each step the system took: what it retrieved and from where, which tools it called and with what arguments, any reasoning text or summary the model returns, and the final output. For an agent, that trace is the difference between “it failed” and “it failed because step three called the wrong tool with stale data.”

Why production demands it

AI systems are non-deterministic and fail silently. The same prompt can take a different path tomorrow, models and data drift over time, and a hallucinated answer carries the same confidence as a correct one. Observability is how you keep up: it lets you debug specific failures, watch quality trend in real time, and trust that the system still works.

What a trace looks like

Most AI observability tools borrow their structure from distributed tracing. In OpenTelemetry’s model, a trace is the path of one request through an application, and it is built from spans: units of work, each with start and end times and attributes, and each linked to a parent span unless it is the root. For an agent, the root span is the user’s request, and child spans record each model call, retrieval, tool call and handoff, with the prompt, the arguments and the result attached. OpenTelemetry now maintains semantic conventions for generative AI, including spans for model clients and MCP, so traces from different frameworks can share field names instead of each vendor inventing its own. The LLM tracing entry covers the format in more detail.

Where observability stops and evaluation starts

A trace tells you what happened. It doesn’t tell you whether the answer was good. That judgment comes from evals: scoring outputs against a rubric, a reference answer or an LLM judge. The two work as a loop. Traces from production supply the real failures, and those failures become test cases, which is how tools such as MLflow Tracing present it: traces show where an issue occurred beneath the framework’s abstractions and feed evaluation datasets. Observability vs. evaluation vs. benchmarking sets out what each one answers.

What to record from day one

Start with the inputs and outputs of every model call, the retrieved documents and their sources, every tool call with its arguments and response, latency and token counts per step, and the model and prompt version that produced each answer. The versions matter most when something changes: without them you can’t tell whether a quality drop came from a new model, a prompt edit or drift in the data. Logs of prompts and outputs can hold personal or confidential data, so decide what to redact and how long to keep traces before you turn recording on.