Skip to content

AI Glossary · Metrics

What is AI Benchmark?

AI Benchmark

An AI benchmark is a standardized test set used to compare models on a task — same questions, same scoring, so results are comparable across models. A public benchmark compares models on its own tasks and settings; it doesn't tell you how a model performs on your specific use case, which is what your own evaluation is for.

Also known as: benchmark, AI benchmarks, model benchmark

· Updated · Chain of Thought

AI Evaluation & Reliability

  • 2 explainers
  • 8 episodes
  • 3 sources

A benchmark is a fixed test — a shared set of questions and a scoring method — that lets you put two models side by side: MMLU for knowledge, HumanEval for code, and so on. It’s how the field tracks general progress and how model releases get compared on a level field.

The trap is reading a benchmark score as a guarantee. A model that tops a leaderboard can still fail on your data, your formats, and your edge cases — and benchmarks can leak into training data, inflating scores. That’s the distinction the show keeps drawing: benchmarks measure general capability, evaluation measures whether the system works for your task. Use benchmarks to shortlist models; use your own evals to decide what ships.

Same model, different score

The settings are part of the score. In the HumanEval paper, the Codex model solves 28.8% of the problems when it generates one sample per problem. Allowed 100 samples per problem, with a problem counted as solved if any sample passes its unit tests, the same model solves 70.2%. Nothing about the model changed; the attempt budget did.

Before comparing two published numbers, check that they share the dataset version, the prompt and number of examples, the sampling settings, the number of attempts and the scoring rule. EleutherAI’s evaluation harness exposes those settings for exactly this reason. If they differ, the scores measure different things.

Sources

Go deeper

  • EleutherAI: evaluation harness CLI docs

    Compare task, few-shot, model and generation settings before interpreting two scores. Running examples requires Python, model access and compute.

From the conversation