Skip to content

Hamel Husain

Parlance Labs

  • 1 episode on Chain of Thought
  • 3 topics covered
  • 1 full transcript
  • Aug 2025 first on the show
Hamel Husain

At the time of his Chain of Thought conversation, Hamel Husain was an independent AI consultant with experience at Airbnb and GitHub. He has since joined OpenAI. On the show he explained why vanity dashboards and a "buffet of metrics" give teams a false sense of security, and why disciplined error analysis grounded in looking at real data is what actually improves AI.

Course: AI Evals for Engineers & PMs, taught with Shreya Shankar.

Cover art for Mindset Over Metrics: How to Approach AI Engineering | Hamel Husain

Latest episode with Hamel Husain · EP 37

Mindset Over Metrics: How to Approach AI Engineering | Hamel Husain

Hamel Husain, Parlance Labs ·

Episodes

What the conversations cover

These notes summarize the episodes featuring Hamel Husain, including other speakers in panel conversations. Follow a source link to read the discussion in context.

EP 37 Mindset Over Metrics: How to Approach AI Engineering | Hamel Husain

  • The first question Hamel Husain gets is “What are the tools?” — and he says that’s the wrong question. No tool abstracts away whether your AI is doing the right thing for the user; until something like AGI arrives, that’s not possible. The right question is “What’s the right process?” because every tool still forces you through one. Episode takeaways
  • Generic dashboard metrics — hallucination score, toxicity score, conciseness score — are, in Hamel’s words, “most of the time not helpful at all.” They’re too generic, don’t necessarily correlate with actual failures in your product, and create an illusion you’ve checked the evals box while you’re really monitoring nothing. Episode takeaways
  • Hamel grounds evals in failures through error analysis: open a trace viewer, read traces, take notes on what’s going wrong, then categorize and count those error types to decide what to prioritize. He notes the technique predates machine learning — it comes from the social sciences — and most people skip it because no one taught them to do it. Episode takeaways

In their words

… You'll be surprised, like, how many people think prompt is an abstract concept because they say something in a meeting and the expectation is a developer's gonna write the prompt. That's the worst thing that can possibly happen. So whatever way possible you needed to get away from that.

Hamel Husain Mindset Over Metrics: How to Approach AI Engineering | Hamel Husain · 36:58

Transcript excerpts; ellipses mark omitted words. Some automatically selected quotes have filler words removed.