Benchmark Saturation

Think of a school exam where every top student scores 98 or 99. The exam can no longer show who is strongest, and a half-point gain is likely noise. Popular AI benchmarks go the same way as models improve. The authors of Humanity’s Last Exam point to MMLU, a multiple-choice knowledge test, where leading models now score above 90%. Their response, like most responses to saturation, was to build a harder test, which can saturate in turn.

Saturation is a different problem from benchmark contamination, where test questions leak into training data, though both make a score look more impressive than it is. A high score on a nearly saturated test can reflect real capability, leakage or models tuned to that particular test, and the number alone cannot tell you which. For a decision about your own product, a saturated public benchmark is at most a shortlist filter; your own evaluation has to do the rest.

Saturation depends on the kind of task. In episode 57 at 15:01, Snorkel AI’s Alex Ratner says few public benchmarks test multi-step tasks as long as real-world work, and “most of them are very much unsaturated.” On his account, short single-answer tests run out of headroom first, while long-horizon agent tasks still have plenty. Galileo was the episode’s presenting sponsor; the read is at 1:36.

In episode 54 at 16:41, Dan Klein of Scaled Cognition compares benchmarks with car crash tests: most cars pass with flying colors because makers have “optimized to make sure that the tests are covered,” which does not mean the underlying safety issues are solved. A new test makes most of them fail again. Galileo was the episode’s presenting sponsor; the read is at 2:53.