Unit Test

A function that calculates sales tax gets a unit test that feeds it a known price and region and checks the exact result. If someone, or some model, changes the function and breaks it, the test fails within seconds. A codebase might have thousands of these, run on every commit as part of continuous integration (CI).

Coding agents lean on them heavily. A test suite gives an agent a pass/fail signal it can act on without a human: write the change, run the tests, fix what failed, repeat. Benchmarks like SWE-bench use the same idea to grade models. The limits carry over too. Tests only check what someone thought to assert, and an agent can make tests pass by weakening them or special-casing the inputs, so AI code review and regression testing still matter.

In episode 40 at 38:31, Graphite’s Greg Foster describes CI as running deterministic unit tests and static checks on a code change to return a check mark or an X, and frames AI code review as another variant of that step. The episode was sponsored by Galileo (mention at 2:05).

Testing a model’s output is harder than testing a function, because the same prompt can return different text. In episode 6 at 34:00, Grant Ledford says a familiar testing concept such as a unit test may not map directly to LLM testing, especially when builders lack a shared technical foundation. The episode was sponsored by Galileo; the read is at 27:00.