Eval-Driven Development

A team building a customer-support bot wants it to stop promising refunds it can’t give. Before touching the prompt, they collect 40 real conversations where that went wrong, write a check for each (a rule or an LLM-as-a-judge rubric), and confirm the current version fails them. Then they change the prompt, rerun the whole suite, and ship only if the new cases pass and older ones still do. Each later bug becomes another eval.

The loop is borrowed from test-driven development, but the checks are different. Model output varies from run to run, so evals usually score many examples and compare pass rates, not a single exact answer. They also need maintenance: an eval set built from last quarter’s traffic can miss what users ask now. Keeping a fixed set as a regression test and refreshing a separate set from production traces covers both.

On the show the term comes up as a practice guests wish were more common. In episode 38 at 22:05, Aurimas Griciūnas calls evals “an old topic” that is still a gap: not everyone has adopted eval-driven development, though he considers it crucial. The episode was sponsored by Galileo, which Conor plugs at 28:13.

Episode 13 is titled after “evaluation-driven development”. In it, at 13:44, Galileo CEO Vikram Chatterji describes the workflow: find the failure modes, build evals around them, put them in the CI/CD pipeline, and keep checking after release. He is describing what his company sells, and the episode was sponsored by Galileo; he is introduced as its CEO at 0:01.

See AI evaluation for what an eval is, and spec-driven development for the related idea of writing intent down before an agent writes code.