A team tweaks its assistant’s prompt so it stops writing overly long answers. The change works, but it also makes the assistant drop the safety warning it used to add to medication questions. A regression suite catches that before release: 300 saved cases, each with a check, run automatically on every change, with the pass rate compared against the last release.
In ordinary software, a regression test either passes or fails. AI outputs vary from run to run, so AI regression suites usually score each case with a rule, a reference answer or an LLM-as-a-judge, and compare aggregate pass rates within a tolerance. Some cases are run several times to separate a real regression from noise. Teams also track cost, latency and token use on the same suite, because a model swap can hold quality steady while doubling the bill.
The suite should be stable on purpose. Add a case every time a bug reaches users, but don’t rewrite old cases to match new behavior without a reason. Keep a separate, refreshed set for measuring improvement, as eval-driven development does. Provider-side model updates count as changes too, so rerun the suite when a hosted model version changes.
In episode 13 at 18:32, Galileo CEO Vikram Chatterji says engineers building with models ask what a unit test or a regression test means for them now, and that evals are the answer. He makes the same point in episode 46 at 34:41, describing eval engineering as what lets teams do proper regression testing inside CI/CD. Galileo sells evaluation tooling. It sponsored both episodes: Chatterji is introduced as its CEO in episode 13 at 0:01, and Conor announces its season-three sponsorship in episode 46 at 1:08.
See AI evaluation for building the cases, and canary deployment for checking a change on live traffic after it passes.
Hear it from the guest
“What is a unit test for me now? And what is a regression test for me now? And that's where eval has come in.”
Quotes lightly edited to remove filler words.
Sources
- Anthropic: Demystifying evals for AI agents — Says Descript's video-agent team runs separate suites for quality benchmarking and regression testing, and that once evals exist, teams get baselines and regression tests for latency, token usage, cost per task and error rates; checked October 24, 2026.
- OpenAI: Evaluation best practices — Recommends continuous evaluation that runs evals on every change and grows the eval set over time; checked October 24, 2026.
