Token price is one input. An agent can make several model calls, query paid tools, retry failures, and require human review. Our recommendation is to choose an explicit accounting boundary and compare the same costs across candidate systems.
Worked example: 100 attempted tasks cost $20 in model and tool charges, and 80 succeed. Cost per attempt is $0.20; cost per success is $0.25. Both are valid measurements, but they answer different questions. Human review would need an additional cost estimate if it is part of the comparison.
In episode 64, Jiaona Zhang argues for measuring returned time and useful outcomes instead of rewarding token consumption. Conor’s Intercom write-up offers a case study of specializing one pipeline step. Neither gives a universal cost target: the task’s value and acceptable failure rate still matter. Plotting cost per success against quality for each candidate shows its Pareto frontier. Budget for abuse as well: model denial of service attacks aim to drive up the cost of each request.
Sources
- Episode 64: Jiaona Zhang on useful AI work — Primary interview motivating outcome-based accounting; the arithmetic example is illustrative.
Go deeper
From the conversation
-
Stop Token Maxxing: Find Where AI Actually Pays Off | Jiaona Zhang, Laurel -
How Intercom Cut $250K/Month by Ditching GPT for Open Models | Fergal Reid -
Writing Code Was Never the Bottleneck | CloudBees CEO Moritz Plassnig -
This is the worst AI tutor we'll ever have | DataCamp -
Agents Are Overtaking Humans in Search: Why Search Must Be Rebuilt -
Most of the Web Will Never Get APIs for AI Agents | Dhruv Batra, Yutori