Most traditional unit tests check that a given input produces an exact expected output. An LLM or agent can give a different, plausible-sounding answer every time, which breaks that model. AI evaluation fills the gap: a repeatable way to score outputs against criteria — accuracy, relevance, safety, format, task completion — so you know whether a change made the system better or worse.
In practice it spans offline evals on a fixed test set (catch regressions before you ship) and online evals on live traffic (catch what production surfaces). The hard part is defining “good” for open-ended outputs, which is why methods like LLM-as-a-judge and human review both show up, and why observability and evaluation get used together rather than confused.
It’s a recurring theme on the show because it’s where most AI projects fail silently: teams ship on vibes, can’t tell why quality drifts, and have no way to improve systematically. Evaluation is the feedback loop that makes iteration possible — without it you’re guessing.
Check the effect, not the claim
An example: a support assistant has to update a ticket and explain the next step to the customer. Grade those separately. For the update, inspect the ticket’s actual state; the assistant saying “updated” does not show the write happened. For the explanation, judge it against the policy. Repeat important cases, since outputs vary, and record the application, dataset and grader versions with each result so a later change in score can be traced.
Sources
- Anthropic: Define success criteria and build evaluations — Recommends evals that mirror the real task distribution, including edge cases, and grading that is automated where possible, by string match, code or a model; checked October 26, 2026.
Go deeper
- How do you evaluate an AI agent? AI, decoded · How to Evaluate an AI Agent: Steps, Outcomes and Side Effects
- What's the difference between AI observability, evaluation, and benchmarking? AI, decoded · Observability vs. Evaluation vs. Benchmarking
- How do you test an AI system when the output isn't deterministic? AI, decoded · How to Test an AI System
- Should you evaluate AI with an LLM-as-a-judge or with human review? AI, decoded · LLM-as-a-Judge vs. Human Evaluation: When to Use Which
- What are AI evals, and what should leaders know about them? AI, decoded · AI Evals: A Guide for Leaders
- Inspect: evaluation tutorial docs
Follow a dataset, solver and scorer through an evaluation, then inspect its logs. Requires Python and model access; live model calls may incur costs.
From the conversation
-
Explaining Eval Engineering | Galileo's Vikram Chatterji -
Mindset Over Metrics: How to Approach AI Engineering | Hamel Husain -
Practical Lessons for GenAI Evals | Chip Huyen & Vivienne Zhang -
Agents Are Overtaking Humans in Search: Why Search Must Be Rebuilt -
The 2025 AI Shift: From Chat to Task Completion & Reliable Action | Galileo Founders -
Beyond Chatbots: How Twilio Uses AI to Strengthen Human Connection | Vinnie Giarrusso -
Architecting AI Agents: The Shift from Models to Systems | Aishwarya Srinivasan -
Why Enterprises Need a Different Approach to AI Agents | Lyzr’s Siva Surendira