Best Chain of Thought Episodes on AI Evaluation
Measuring whether AI works: evals, reliability, trust.
Alex Ratner opens with the gap between agent capabilities and the evaluations used to measure them. The collection then moves through evaluation as an engineering discipline, agent reliability and evaluation-driven development. Chip Huyen and Vivienne Zhang close with practical lessons for building an evaluation process.
- 5 episodes
- 6 guests
- 3 hours of listening
- 2 jump-to moments
- 1
Every AI Agent Has an Evaluation Gap | Alex Ratner, Snorkel AI Alex Ratner, Snorkel AI Alex Ratner on how agent capabilities have outpaced evaluation, and the benchmarks Snorkel is building to measure them. Jump to the moment · 3:09 · Capability outpaces measurement → - 2
Explaining Eval Engineering | Galileo's Vikram Chatterji Vikram Chatterji, Galileo Vikram Chatterji on treating evaluation as an engineering discipline, not a one-time check. Jump to the moment · 3:24 · Author, test, and measure evals → - 3
The AI Agent Trust Gap: Bridging Risk to Reliability | Elastic’s Philipp Krenn Philipp Krenn, Elastic Philipp Krenn on moving agents from risky to reliable, and why trust is the bar for shipping them. - 4
How Engineering Leaders Can Future-Proof Their AI Strategy | Vikram Chatterji on Dev Interrupted Vikram Chatterji & Andrew Zigler The case for evaluation-driven development: build the eval loop into how you ship instead of adding it afterward. - 5
Practical Lessons for GenAI Evals | Chip Huyen & Vivienne Zhang Chip Huyen & Vivienne Zhang Chip Huyen and Vivienne Zhang on practical lessons for GenAI evaluation.
- Best Chain of Thought Episodes for AI Founders Building an AI company: defensibility, GTM, and where the market stands.
- Best Chain of Thought Episodes for Engineers Shipping with AI: the craft, the workflow, and where it falls short.
- Best Chain of Thought Episodes on AI Agents How agents work: architecture, memory, context, frameworks.
- Best Chain of Thought Episodes on Agent Memory How agents remember: memory stores, context engineering, and what breaks without them.
- Best Chain of Thought Episodes on Multi-Agent Systems Many agents, one system: orchestration patterns, protocols, and the overhead they add.
- Best Chain of Thought Episodes on Open Source AI Open weights, open ecosystems, and what they cost in production.
- Best Chain of Thought Episodes on AI Coding Agents What changes downstream when agents write most of the code.
- Best Chain of Thought Episodes for Enterprise AI Leaders Getting AI past the pilot: ROI, rollout, and the data underneath it.
- Best Chain of Thought Episodes on AI Security Attacking agents, defending them, and the part software cannot cover.
- Best Chain of Thought Episodes on RAG & Retrieval Getting the right context to the model: chunking, vectors, graphs, and what breaks at scale.
- Best Chain of Thought Episodes on AI Infrastructure & Hardware What AI runs on: power, GPUs, networks, kernels and small models.
- Best Chain of Thought Episodes on Building AI Products Deciding what to build, setting a quality bar and shipping fast.