LLM observability tools compared: choose where traces go

Write down the question a trace must answer

A trace is useful when someone can read it and explain a bad answer. Pick one recent failure before you open a vendor demo. A support agent quoted an old refund policy, say. Your trace has to show which documents retrieval returned, which tool the agent called with which arguments, which model and prompt version produced the reply, and how long each step took. If the tool you are testing cannot show those four things for that request, its dashboards will not help much.

The AI observability explainer covers why ordinary request monitoring misses these failures. This guide covers the product decision. The documented workflows in the table come from the linked primary documentation, checked October 24, 2026. The fit questions and the pilot are our editorial recommendations. The five options illustrate different starting points; they are not a complete inventory or a ranking, and this guide reports no measured performance.

Separate the instrumentation from the backend

Two decisions hide inside “choose an observability tool”. The first is how spans leave your code: a vendor SDK, a framework integration, OpenTelemetry instrumentation or a gateway that sees each model request. The second is where those spans are stored and who can query them. Keep them apart. Vendor-specific instrumentation can be quick to set up, and it makes a later move to a different backend more expensive.

OpenTelemetry GenAI conventions: the GenAI semantic conventions repository defines spans, metrics and events for model clients, agents and MCP, plus provider-specific conventions. The project moved these conventions out of the main OpenTelemetry documentation into this repository, and its pages mark them with Development status, so attribute names can still change. The conventions define a format; you still need a backend to store and query the spans. They also describe options for recording prompt and response content on spans or uploading it to external storage. Decide which option your privacy rules allow before you turn content capture on.

Langfuse: its observability overview describes traces that capture prompts, responses, token usage, latency and tool or retrieval steps, plus cost tracking, scores, dashboards and alerts. Langfuse also has a native OpenTelemetry integration. According to the self-hosting documentation, a deployment runs with Docker Compose or Kubernetes on Postgres, ClickHouse, Redis or Valkey, and S3-compatible storage, and some add-on features need a license key. The repository license is MIT outside its enterprise directories. Count those four services in the operating cost if you plan to self-host.

LangSmith: the observability documentation covers tracing, dashboards, alerts, automations and feedback collection, and it lists integrations beyond LangChain. LangSmith also accepts traces over OpenTelemetry. The self-hosting page describes self-hosted LangSmith as an add-on to the Enterprise plan, and the pricing page lists monthly base-trace allowances by plan. If trace data must stay inside your network, confirm which plan allows that before you run the pilot.

Helicone: the quick start routes requests through an OpenAI-compatible gateway by changing the client’s base URL, then shows each request in a dashboard. The repository is Apache 2.0 licensed, documents a Docker Compose self-hosting path, and lists sessions for tracing agents. A gateway sees what passes through it. A retrieval query or a tool call that never reaches the gateway needs session metadata or separate instrumentation to show up in the same trace.

Phoenix: its tracing documentation accepts traces over OTLP and lists OpenInference instrumentation for frameworks such as LlamaIndex, LangChain and DSPy and for several model providers. The Phoenix overview describes how those traces feed evaluation, prompt iteration and experiments. The repository license is the Elastic License 2.0, which bars offering the software to others as a hosted or managed service. That restriction rarely affects internal use. Check it if you build a product on top.

We left Galileo out of this comparison. Conor worked at Galileo, and Galileo sponsored the show’s first season, most of its second and the first fifteen episodes of its third.

Run a trace-completeness pilot

Suggested pilot, not a reported benchmark: instrument one real workflow with each tool on your shortlist. Use a staging copy with redacted data, then send the same requests through every setup.

Pilot request	What the trace has to show
Retrieval returns an outdated document	The query, the documents returned with their order and scores, and the document the answer used
A tool times out and the agent retries	Both attempts, their arguments and timing, and which result the agent kept
A prompt or model changes between two runs	The prompt version and model on each span, so you can compare the runs side by side
One request fans out to two agents	A single trace with parent and child spans, not two unrelated records
A request contains personal data	The redaction or content-capture setting you chose, applied before storage
The tracing backend is unreachable	The application keeps serving, and you know which spans were dropped or delayed

Run each request once with a deliberate mistake as well. Point retrieval at the wrong index, or remove a tool’s instrumentation. If the trace looks the same either way, the setup is not capturing the step you care about. To turn a captured failure into a regression test, continue with the AI evaluation tools comparison.

Count the data you will keep

Traces of model calls can hold full prompts, retrieved documents and customer messages. Decide what to sample, what to redact and how long to retain it, then confirm each vendor’s settings match. Ask what an export contains and whether you can re-send stored traces to a second backend. Measure the latency your instrumentation adds and how ingestion behaves at peak load. Keep your OpenTelemetry instrumentation separate from any one vendor’s SDK where you can, so changing the backend later is a configuration change rather than a rewrite.

Where the interviews fit

Both cited guests describe OpenTelemetry as the common layer. Redpanda CTO Tyler Akidau said his team accepts agent traces from any framework that emits it. Elastic’s Philipp Krenn compared OpenTelemetry’s role for telemetry to MCP’s role for AI tools. These are the guests’ views, and they predate some of the documentation above. Episode 32 was sponsored by Galileo while Conor worked there, and Conor asks Krenn how Galileo and Elastic work together just before the quoted passage.