MMLU works like a very wide school exam. Each question has four answer choices, and the subjects range from high school chemistry and US history to professional law, medicine and accounting. A model’s score is the share it gets right, overall and per subject. Random guessing gets about 25%.
When Dan Hendrycks and colleagues released it in 2020, most models scored close to chance and the largest GPT-3 model reached roughly 44%. Because it was broad, easy to run and hard at the time, MMLU became the headline number in model launch announcements for several years. Then models caught up. By the time the Humanity’s Last Exam authors wrote in 2025, frontier models scored over 90%, so differences between top models were small and the test told you little about which was better. Harder successors such as MMLU-Pro, with more answer choices and more reasoning-heavy questions, followed.
MMLU also shows the limits of a static AI benchmark. Its questions are public, so they can appear in training data, the benchmark contamination problem. A 2024 audit found that some questions have wrong answer keys, with errors in 57% of the sampled virology questions. And multiple-choice knowledge is not the same as doing useful work. MMLU is still a reasonable quick check that a smaller or open-weights model has broad knowledge, but it should not decide which model to deploy.
Sources
- Hendrycks et al.: Measuring Massive Multitask Language Understanding (2020) — Introduces the test, covering 57 tasks including elementary mathematics, US history, computer science and law, and reports that most models of the time scored near random chance; checked October 24, 2026.
- MMLU dataset and code (GitHub) — The authors' official repository with the questions, evaluation code and a table of published model results; checked October 24, 2026.
- Gema et al.: Are We Done with MMLU? (2024) — Finds numerous ground-truth errors in MMLU, including errors in 57% of the analysed Virology questions, and releases the corrected MMLU-Redux subset; checked October 24, 2026.
- Phan et al.: Humanity's Last Exam (2025) — States that LLMs now achieve over 90% accuracy on popular benchmarks like MMLU, limiting its usefulness at the frontier; checked October 24, 2026.