By 2024, frontier models scored over 90% on MMLU and similar tests, so those benchmarks no longer separated the best models. Humanity’s Last Exam was built to restore that headroom. The Center for AI Safety and Scale AI collected questions from subject-matter experts; the paper describes them as unambiguous and easily verifiable, but not answerable with a quick internet search. The final set has 2,500 questions across more than a hundred subjects, from mathematics to classics. Some include images, and each has a single verifiable answer, either multiple choice or an exact short answer.
When it launched in early 2025, the paper reported that state-of-the-art models showed low accuracy and poor calibration on it: they were often wrong while sounding confident. Scores have risen since, so treat any specific number as a snapshot and check the date. The organizers keep a private held-out set to detect overfitting to the public questions, and released HLE-Rolling, a dynamic version, in October 2025.
The name is a claim about closed-ended academic questions, not about AI in general. A high HLE score shows a model can answer hard expert questions with checkable answers. It does not show that the model can do open-ended research, carry out multi-step work as an AI agent, or know when it lacks information. Like other AI benchmarks, it is likely to saturate eventually, and contamination of public questions into training data, benchmark contamination, is a standing risk.
Sources
- Phan et al.: Humanity's Last Exam (2025) — Introduces HLE as a multimodal benchmark at the frontier of human knowledge, designed to be the final closed-ended academic benchmark of its kind, created because LLMs exceed 90% on benchmarks like MMLU; checked October 24, 2026.
- Humanity's Last Exam official site — Official project page: HLE was finalized with 2,500 questions across over a hundred subjects, a private held-out set is kept to check for overfitting, and a dynamic fork, HLE-Rolling, was released in October 2025; checked October 24, 2026.