LLM Leaderboard

A team picking a model for a new feature often starts by checking a leaderboard, the AI version of a league table. There are two main kinds. Benchmark leaderboards run every model on the same fixed tests, such as MMLU or SWE-bench, and sort by score. Preference leaderboards ask people which answer they like better. Arena, formerly LMArena and Chatbot Arena, shows a visitor two anonymous models’ answers to the same prompt, asks which is better, and turns the votes into a rating, much as chess ratings are built from game results. That is pairwise evaluation at crowd scale.

Leaderboards are useful for spotting which models are worth testing, but a high rank does not mean a model is best for your task. Preference votes reward answers people find pleasing, which can favor length and style over accuracy. Public tests can leak into training data, a problem called benchmark contamination. And rankings can be gamed: the 2025 “Leaderboard Illusion” paper reported that some providers privately tested many model variants on Chatbot Arena and published only the best, which tilted the rankings in their favor.

The practical lesson is to treat a leaderboard as a shortlist and then run your own AI evaluation on a golden dataset from your product. An AI benchmark measures one thing well; a leaderboard compresses many such measurements, or many people’s tastes, into one number.

In episode 27 at 25:51, Cohesity’s Greg Statton says data work is “far less exciting than saying, well, I want to go test out” the “top of the leaderboard language model,” but that a top model still returns garbage if it is given garbage data. Galileo sponsored the episode, and Conor mentions working with Galileo’s enterprise customers later in it.