Skip to content

Alex Ratner

Snorkel AI

  • 1 episode on Chain of Thought
  • 5 topics covered
  • 1 full transcript
  • Apr 2026 first on the show
Alex Ratner

Alex Ratner is the co-founder and CEO of Snorkel AI, which he started out of Chris Ré’s Stanford lab while helping establish data-centric AI as a field. On Chain of Thought he mapped the evaluation gap keeping enterprise agents stuck in demos and walked through Snorkel’s Open Benchmarks grant program.

Cover art for Every AI Agent Has an Evaluation Gap | Alex Ratner, Snorkel AI

Latest episode with Alex Ratner · EP 57

Every AI Agent Has an Evaluation Gap | Alex Ratner, Snorkel AI

Alex Ratner, Snorkel AI ·

Show notes & transcript → All listening options →

Episodes

What the conversations cover

These notes summarize the episodes featuring Alex Ratner, including other speakers in panel conversations. Follow a source link to read the discussion in context.

EP 57 Every AI Agent Has an Evaluation Gap | Alex Ratner, Snorkel AI

  • Alex Ratner frames the evaluation gap as a hypothesis that AI capabilities have advanced faster than our ability to define and measure them precisely. Trying an agent successfully is different from establishing where it will work reliably. Transcript at 3:09
  • Ratner separates evaluation into input or environment complexity, autonomy horizon, and output complexity. A multi-turn task with changing requirements and a long cited report needs different checks from a single-answer question. Transcript at 15:01
  • Ratner argues that agent environments and datasets often belong together. When an agent must interact with an environment, evaluating it requires more than the question-and-answer payload used for a static test. Transcript at 35:17

In their words

If you want your model to do something different, you don't go and perform brain surgery on the neurons. You don't reinvent the architecture of the algorithm. Let me go and invent a new post-transformer architecture. No, you get data.

Alex Ratner Every AI Agent Has an Evaluation Gap | Alex Ratner, Snorkel AI · 0:00

Transcript excerpts; ellipses mark omitted words. Some automatically selected quotes have filler words removed.