How do you evaluate an AI agent?
You check it at three levels: the step, the turn, and the session. Did it pick the right tool, did it do the steps in the right order, and did the whole thing reach the right result. A single accuracy score hides all of this, which is why agents that look fine in a demo fail in production. Evaluating an agent is less about one number and more about defining, at each level, what 'good' actually looks like for your use case.
AI Evaluation & ReliabilityAI Agents
Step level: did it choose the right tool?
At each individual point in the workflow, did the agent pick the correct tool and use it correctly, with the right parameters. Most agent failures start here: it calls the wrong tool, or the right tool the wrong way.
Turn level: did it do the steps in the right order?
Zoom out one level. Were the steps performed in the correct sequence to reach the correct conclusion. An agent can make every individual call correctly and still fail because it did them out of order, or looped without stopping.
Session level: was the final result actually right?
The whole workflow, end to end. Did the agent deliver the outcome the user actually wanted. This is the level that matters to the person using it, and the one a step-level score can completely miss.
The trap to avoid
A “correct” answer isn’t always a “good” answer. A job-fit bot that tells a candidate “you’re a terrible fit” can be technically correct and completely useless. So evaluating an agent is less about one number and more about defining, at each level, what good actually looks like for your use case.
Why it matters
Only a small share of teams building agents are evaluating them well. The ones who ship reliably are the ones who stopped treating evaluation as a single score and started checking step, turn, and session separately.
From the conversation
This explainer is drawn from these episodes — each carries its full transcript.
