Ask someone to score an email draft from 1 to 10 and you get a different number every time. Show them two drafts and ask which one they would send, and the answer is much more consistent. Pairwise evaluation uses that. A team testing a new prompt runs the old and new versions on the same 300 inputs, shows each pair to a reviewer or an LLM judge, and reports a win rate: the new version won 58% of comparisons, lost 30% and tied 12%.
Many preference votes can be combined into a ranking. The public LLM leaderboard Arena (formerly LMArena and Chatbot Arena) does this with anonymous model pairs and crowd votes. The same preference data trains a reward model in RLHF and is used directly by DPO.
A win rate tells you which output people prefer, not whether either is correct. Two wrong answers can still produce a winner. Judges, human or model, also favor the answer shown first and the longer one, so good setups randomize the order, allow ties and check a sample against expert human evaluation.
In episode 5 at 23:00, NVIDIA’s Vivienne Zhang points to open leaderboards that publish A/B answer pairs, where “a lot of humans don’t agree on which answer is better.” Her point is that open-ended tasks have no single right answer, which makes assessment hard. Galileo sponsored season one; Conor opens the episode as a Galileo employee with Galileo CEO Vikram Chatterji.
In episode 75 at 54:29, WisdomAI’s Kapil Chhabra describes a related comparison of time and cost, testing a frontier model against WisdomAI’s own harness, both given the same context. He reports a 3x difference in time to first answer and 3.4x in cost, rather than preference votes about output quality. These are his company’s figures about its own product, and WisdomAI sponsored this interview.
Hear it from the guest
“Where you can find samples of A B answers from models. And you can see actually a lot of humans don't agree on which answer is better”
Quotes lightly edited to remove filler words.
Sources
- Chiang et al.: Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024) — Ranks models with a pairwise comparison approach, crowdsourcing votes between two anonymous model answers to the same prompt; checked October 24, 2026.
- Zheng et al.: Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023) — Studies pairwise comparison by model judges and reports position, verbosity and self-enhancement biases; checked October 24, 2026.
