Skip to content

AI Glossary · Metrics

What is Dynamic Benchmark?

Dynamic Benchmark

A dynamic benchmark is an evaluation whose test items, and where needed their ground-truth answers, are added or replaced over time, through periodic or ongoing updates such as fresh questions about recent events or new adversarial examples. That makes it harder for a system to score well by memorizing a fixed answer set.

· Updated · Chain of Thought

AI Evaluation & Reliability

  • 1 explainer
  • 1 episode
  • 1 guest clip
  • 2 sources

A static benchmark stays fixed after release, so its questions and answers can leak into training data (benchmark contamination) or be tuned against directly. A dynamic benchmark changes the test instead. LiveBench adds and updates questions every month from recent sources such as math competitions, arXiv papers and news articles, and scores answers against objective ground truth. Dynabench takes a different route: annotators write examples that the current model gets wrong but a person would not, so each round targets what models still miss. Refreshing the questions reduces the contamination risk without eliminating it.

On Chain of Thought, Saahil Jain of You.com, which sells a search API for AI agents, said many search benchmarks don’t correlate with real-world performance, and that a static set lets a company “essentially memorize the answer” (episode 81, 50:27). Scoring well that way without generalizing, he said, leads to a distribution shift in performance once the system meets questions outside the benchmark (52:00). He described an internal You.com benchmark that changes over time, which the company plans to release, and one built with a partner that changes even more often, with ground truth arriving hours rather than days later (52:00, 52:26). Because You.com counts freshness as one of the core capabilities of a search API, he said, it wants benchmarks that force its APIs to stay fresh.

The practical decision is not to discard static benchmarks. Jain said they remain useful for checking that a model’s reasoning works well with your tool, though it is “quite possible to overfit on them” (52:52). His advice was to choose benchmarks aligned with the capabilities you care about, then validate with customers using the product at scale (53:17) and with custom evals built for specific companies, since “a benchmark is simply a way of simulating what a company would care about” (53:41). Those are claims from one vendor’s experience, not a settled standard. Episode 81 credits Inngest and Walrus Memory as presenting sponsors and CoralBricks and WisdomAI as episode sponsors (sponsor thank-you); the WisdomAI read, about its own benchmark study, runs just before this exchange (49:29).

Worked example: a search tool scores 92% on a fixed set of 500 news questions written last year. The team also builds a fresh set each week: 200 questions about events from the previous 48 hours, with answers checked against published sources after the questions are written. On this week’s set the same system scores 71%. The 21-point gap suggests the static score overstates how well the tool handles current information, but not why: the fresh questions may be harder, or the index may lag. Compare systems on the same week’s set and report its date with the score. A change from one week to the next mixes system changes with question changes, so rerun the previous version on the new set before claiming an improvement.

AI benchmark covers fixed test sets in general, benchmark saturation covers sets that top systems have stopped separating, and AI evaluation covers testing against your own workload.

Hear it from the guest

“So what we've been thinking a lot about is dynamic benchmarks.”
Watch on YouTube

Quotes lightly edited to remove filler words.

Sources

Go deeper

From the conversation