A benchmark is a fixed test — a shared set of questions and a scoring method — that lets you put two models side by side: MMLU for knowledge, HumanEval for code, and so on. It’s how the field tracks general progress and how model releases get compared on a level field.
The trap is reading a benchmark score as a guarantee. A model that tops a leaderboard can still fail on your data, your formats, and your edge cases — and benchmarks can leak into training data, inflating scores. That’s the distinction the show keeps drawing: benchmarks measure general capability, evaluation measures whether the system works for your task. Use benchmarks to shortlist models; use your own evals to decide what ships.
Same model, different score
The settings are part of the score. In the HumanEval paper, the Codex model solves 28.8% of the problems when it generates one sample per problem. Allowed 100 samples per problem, with a problem counted as solved if any sample passes its unit tests, the same model solves 70.2%. Nothing about the model changed; the attempt budget did.
Before comparing two published numbers, check that they share the dataset version, the prompt and number of examples, the sampling settings, the number of attempts and the scoring rule. EleutherAI’s evaluation harness exposes those settings for exactly this reason. If they differ, the scores measure different things.
Sources
- Hendrycks et al.: Measuring Massive Multitask Language Understanding (MMLU) — Introduces MMLU, a test of multitask accuracy across 57 subjects including elementary mathematics, US history, computer science and law; checked October 25, 2026.
- Chen et al.: Evaluating Large Language Models Trained on Code (HumanEval) — Releases HumanEval, which measures functional correctness of programs synthesized from docstrings. Codex solves 28.8% of problems with one sample each and 70.2% with 100 samples per problem; checked November 7, 2026.
- Deng et al.: Investigating Data Contamination in Modern Benchmarks for Large Language Models — Examines the gap between inflated benchmark scores and real performance caused by evaluation data leaking into training; checked October 25, 2026.
Go deeper
- What's the difference between AI observability, evaluation, and benchmarking? AI, decoded · Observability vs. Evaluation vs. Benchmarking
- Can switching to an open model actually cut your AI costs? AI, decoded · When an Open Model Beats a Frontier API on Cost
- EleutherAI: evaluation harness CLI docs
Compare task, few-shot, model and generation settings before interpreting two scores. Running examples requires Python, model access and compute.
From the conversation
-
Agents Are Overtaking Humans in Search: Why Search Must Be Rebuilt -
Every AI Agent Has an Evaluation Gap | Alex Ratner, Snorkel AI -
This is the worst AI tutor we'll ever have | DataCamp -
Asking Your Team to Automate Their Jobs With AI | Cloudera's Shayde Christian -
Architecting AI Agents: The Shift from Models to Systems | Aishwarya Srinivasan -
How Cisco Secures Personal AI Assistants for 90,000 Employees | DJ Sampath -
Using AI to Modernize Your Legacy Applications | MongoDB’s Rachelle Palmer -
AI's Trillion-Dollar Healthcare Bet | Corti's Andreas Cleve