What should you measure on an AI agent besides accuracy?

In detail

Why accuracy alone misleads

A single accuracy score collapses a multi-step process into one number. It rewards an agent that stumbled into the right answer and punishes one that reasoned well but hit a broken tool. For anything that takes actions rather than producing one response, you need to measure the path, not just the destination. Snorkel AI’s Alex Ratner gives the reason in episode 57 at 6:05: models have crushed math and coding competition problems, yet a coding agent can still struggle with a mundane-seeming task that has messier inputs.

Consistency: does it work every time?

One run can’t tell you whether an agent is reliable, because the same agent can pass a task once and fail it on the next try. The τ-bench benchmark made this measurable. Its authors proposed pass^k, the chance that an agent succeeds on all k independent attempts at the same task, and found that even GPT-4o, then a leading function-calling model, solved under half of the tasks and was quite inconsistent: in the retail domain, pass^8 fell below 25%. That is the measurement behind Dan Klein’s point in episode 54 at 16:41. A system that gets a smaller set of scenarios right every time can be worth more than one with a higher average that varies run to run. Note the contrast with pass@k, which asks whether any one of k attempts succeeds and so rewards a different thing.

Process metrics: how it got there

Watch the steps. Did the agent choose the correct tool, and call it with the right parameters? Did it perform the steps in a sensible order? When a call failed, did it recover or spiral? These catch the failures that an end-result score steps right over. Galileo’s Vikram Chatterji lists them in episode 13 at 15:46: did the agent choose the right tool, and did that tool get called the right way? Chip Huyen’s method in episode 5 at 39:00 starts from the same place: break down how the application is going to fail, then put metrics at each point of failure. For comparing models’ tool calling, the Berkeley Function Calling Leaderboard evaluates call accuracy.

Cost and efficiency

The same task can be done in three steps or thirty. Track steps per task, token and dollar cost, and latency. An agent that gets there but burns ten times the budget is a production problem even when its accuracy looks fine.

Experience and oversight

Over a real conversation, does the agent hold the user’s intent or drift off it? How often does it need a human to take over? Intervention rate and conversation quality are the metrics that tell you whether people can rely on the thing.

Disclosure: Episode 5 was sponsored by Galileo while Conor worked there (read at 20:00). Vikram Chatterji is Galileo’s CEO, and episode 13 was published while Conor worked at Galileo. Episodes 54 and 57 were sponsored by Galileo, the Season 3 presenting sponsor (episode 54 opening read, episode 57 sponsor read).