A static benchmark stays fixed after release, so its questions and answers can leak into training data (benchmark contamination) or be tuned against directly. A dynamic benchmark changes the test instead. LiveBench adds and updates questions every month from recent sources such as math competitions, arXiv papers and news articles, and scores answers against objective ground truth. Dynabench takes a different route: annotators write examples that the current model gets wrong but a person would not, so each round targets what models still miss. Refreshing the questions reduces the contamination risk without eliminating it.
On Chain of Thought, Saahil Jain of You.com, which sells a search API for AI agents, said many search benchmarks don’t correlate with real-world performance, and that a static set lets a company “essentially memorize the answer” (episode 81, 50:27). Scoring well that way without generalizing, he said, leads to a distribution shift in performance once the system meets questions outside the benchmark (52:00). He described an internal You.com benchmark that changes over time, which the company plans to release, and one built with a partner that changes even more often, with ground truth arriving hours rather than days later (52:00, 52:26). Because You.com counts freshness as one of the core capabilities of a search API, he said, it wants benchmarks that force its APIs to stay fresh.
The practical decision is not to discard static benchmarks. Jain said they remain useful for checking that a model’s reasoning works well with your tool, though it is “quite possible to overfit on them” (52:52). His advice was to choose benchmarks aligned with the capabilities you care about, then validate with customers using the product at scale (53:17) and with custom evals built for specific companies, since “a benchmark is simply a way of simulating what a company would care about” (53:41). Those are claims from one vendor’s experience, not a settled standard. Episode 81 credits Inngest and Walrus Memory as presenting sponsors and CoralBricks and WisdomAI as episode sponsors (sponsor thank-you); the WisdomAI read, about its own benchmark study, runs just before this exchange (49:29).
Worked example: a search tool scores 92% on a fixed set of 500 news questions written last year. The team also builds a fresh set each week: 200 questions about events from the previous 48 hours, with answers checked against published sources after the questions are written. On this week’s set the same system scores 71%. The 21-point gap suggests the static score overstates how well the tool handles current information, but not why: the fresh questions may be harder, or the index may lag. Compare systems on the same week’s set and report its date with the score. A change from one week to the next mixes system changes with question changes, so rerun the previous version on the new set before claiming an improvement.
AI benchmark covers fixed test sets in general, benchmark saturation covers sets that top systems have stopped separating, and AI evaluation covers testing against your own workload.
Hear it from the guest
“So what we've been thinking a lot about is dynamic benchmarks.”
Quotes lightly edited to remove filler words.
Sources
- White et al.: LiveBench, a challenging, contamination-limited LLM benchmark — Describes questions drawn from recent math competitions, arXiv papers, news articles and datasets, scored against objective ground truth and added or updated monthly; abstract checked November 7, 2026.
- Kiela et al.: Dynabench, rethinking benchmarking in NLP (NAACL 2021) — Introduces dynamic dataset creation in which annotators write examples a target model misclassifies but a person would not; abstract checked November 7, 2026.