AI Observability
Seeing inside AI systems once they hit production.
AI observability is the practice of instrumenting AI systems so you can see what they actually do in production — tracing each step of an agent or pipeline, collecting telemetry, and surfacing the signals that explain failures and drift.
A trace connects events within one request. OpenTelemetry describes spans with start and end times, attributes and relationships to other spans, so a trace can show how work moves across a system (OpenTelemetry, Traces). In an AI application, use that structure to connect retrieval, model calls and tool execution. Decide which fields are needed to investigate a failure and which data should be redacted before recording it.
Charity Majors in episode 18 describes observability as part of an engineering feedback loop. Vikram Chatterji in episode 46 discusses evaluating the internal steps of agents, including tools and handoffs. What is AI observability? focuses on understanding a request; observability vs evaluation vs benchmarking separates that from judging quality. Read audit trail for the retained record used to reconstruct an action later.
15 episodes
- Parenting Your AI Agents for Best Results | Cisco's Jeetu Patel
- We Built Agents, Nobody Built HR | Tyler Akidau, Redpanda
- Explaining Eval Engineering | Galileo's Vikram Chatterji
- Architecting AI Agents: The Shift from Models to Systems | Aishwarya Srinivasan
- From Demo to Defensibility: How to Build an AI Business that Lasts | Aurimas Griciūnas
- Mindset Over Metrics: How to Approach AI Engineering | Hamel Husain
- Mastering Multi-Agent Systems | MongoDB’s Mikiko Chandrasekhar
- The AI Agent Trust Gap: Bridging Risk to Reliability | Elastic’s Philipp Krenn
- Architecting Reliable Agentic AI | Cisco’s Giovanna Carofiglio on the AGNTCY Collective
- The Emerging AI Agent Stack | CrewAI’s João Moura
- The 2025 AI Shift: From Chat to Task Completion & Reliable Action | Galileo Founders
- AI Won't Solve Your Toughest Engineering Problems | Honeycomb’s Charity Majors
- Inside IBM's watsonx: Building Enterprise AI That Ships | Dr. Maryam Ashoori
- The Enterprise AI Deployment Playbook | ServiceTitan, Indeed & Twilio
- The State of AI: Open-Source Models & Enterprise Trust | May Habib
Explainers on this topic
- AI Agent Metrics Beyond Accuracy
- How to Cut AI Agent Costs
- Observability vs. Evaluation vs. Benchmarking
- What Is AI Observability
- What Is LLM-as-a-Judge
- On-Premise AI for Regulated Enterprises
Terms on this topic
Tool comparisons on this topic
- AI evaluation tools compared: choose your workflow
- LLM observability tools compared: choose where traces go
Guests on this topic
Jeetu PatelTyler AkidauVikram ChatterjiAishwarya SrinivasanAurimas GriciūnasHamel HusainMikiko ChandrasekharPhilipp KrennGiovanna CarofiglioJoão MouraAtindriyo SanyalCharity MajorsMaryam AshooriMehmet Murat EzbiderliVinnie GiarrussoGrant LedfordMay Habib