How do you test an AI system when the output isn't deterministic?

In detail

Why one exact string does not fit every answer

Some tests compare an answer with one exact string. That is too restrictive for flexible language: the same prompt can yield several different answers that are all correct, so assertEqual fails a correct answer just because it’s worded differently from the golden string. Testing AI means checking whether each output has the qualities you require.

Test on properties, against a dataset

Build a dataset of representative inputs paired with what a good answer must do — stay grounded in the source, follow the instruction, stay in scope, avoid the unsafe response. Then score each output against those properties rather than against one golden string. The scoring is done by checks where you can write them and by a rubric or an LLM judge where you can’t. The dataset is the asset; it’s what makes the test repeatable.

Variation is hard to judge without a reference. DevRev’s Manoj Agarwal described early enterprise chatbots in episode 16 at 25:11: companies found that asking the same question three times produced different answers, and “I don’t know which answer was correct. I need to go and ask the subject matter expert now” (25:44). Later he names the double standard: organizations “want perfect answer always” from a machine but accept different answers from people taught the same content (34:10). A reviewed dataset is what turns that inconsistency into something you can score. The episode aired in season two, which Galileo sponsored while Conor worked there; Galileo COO Yash Sheth co-hosted (0:24).

Run it like a test suite

The discipline matters more than any single check. Wire the suite into your workflow so it runs on every prompt change, model swap, and dependency bump, and so a drop in scores blocks the change. This is what turns evaluation from a one-time demo check into the AI equivalent of CI — the thing that catches a regression before it ships.

Check that the suite can fail, too. In a review described in the Chain of Thought newsletter, a reviewer model caught a change that had replaced assertions with console.error() in five checks: they printed “FAILED” for a wrong result and the test still passed. Break one case on purpose and confirm the run goes red before you trust a green one.

Start small and grow the set

You don’t need a thousand cases to begin. Start with the handful that represent your real traffic and the failures you’ve already seen, and add a case every time something breaks in production. The test set should grow toward your actual edge cases, not toward some abstract notion of coverage.

Try it: a minimal property suite

Treat each test case as a list of requirements for the answer. Some are mechanical: cite the supplied source, never use a forbidden phrase, stay under a word limit. Others, like describing the source accurately, need a rubric. A small suite looks like this:

1. Build the first cases from failures. Hamel Husain’s method is error analysis: read traces, take notes on what went wrong, sort the notes into categories, and count which come up most. The top categories become your first cases.
2. Run cheap checks first. Check fields and strings with ordinary code before you pay for a judge. A source-ID check proves the answer names a source; it can’t prove the source supports the claim, so that part goes to the rubric.
3. Gate on a reviewed baseline, and name the failures. Fail the build when the pass rate drops below a baseline stored with the suite, and print which cases failed, because a pass rate alone doesn’t tell an engineer what to fix.
4. Check the judge against people before it gates anything. A judge’s verdict is another model output. If it disagrees with itself across runs, look at the disputed cases before moving the threshold; a gate that flips on noise teaches the team to ignore it.

Illustrative sketch, not production code. Each case in eval_cases.json holds an ID, a prompt, source text, a required source ID, a forbidden phrase, a word limit, and an accuracy rubric. The two stubs stand in for your application and your calibrated judge.

import json
from pathlib import Path


def call_model(prompt, source):
    raise NotImplementedError  # -> {"text", "source_ids"}


def judge_against_rubric(source, answer, rubric):
    raise NotImplementedError  # returns True or False


def test_eval_pass_rate():
    cases = json.loads(Path("eval_cases.json").read_text())
    baseline_file = Path("eval_baseline.json")
    baseline = json.loads(baseline_file.read_text())["pass_rate"]
    assert cases

    failed = []
    for case in cases:
        reply = call_model(case["prompt"], case["source"])
        answer = reply["text"]

        cheap_checks_pass = (
            case["required_source_id"] in reply["source_ids"]
            and case["forbidden_phrase"].lower() not in answer.lower()
            and len(answer.split()) <= case["max_words"]
        )
        # The judge only runs when the cheap checks pass.
        if not (cheap_checks_pass and judge_against_rubric(
            case["source"], answer, case["rubric"]
        )):
            failed.append(case["id"])

    pass_rate = (len(cases) - len(failed)) / len(cases)
    assert pass_rate >= baseline, f"{pass_rate:.0%}, failed: {failed}"

Counting words with split() is coarse; use a stricter method if the limit has a hard product meaning.

The suite is the checkpoint before a change ships, and production shows what it missed. Asked how observability changes for AI-driven systems, Honeycomb’s Charity Majors started from production: “the source of truth is what is happening right now.” (29:44) Her point was that many engineers are comfortable in the editor running tests and treat production as someone else’s problem. When a failure reaches users, trace what the system received and returned, then add the case that would have caught it.

Keep exact checks where the contract is exact

Non-deterministic wording does not make the whole application non-deterministic. A parser can reject malformed JSON. An authorization check can reject a cross-tenant request. A refund record can contain the exact amount. Keep ordinary tests for those boundaries; use property checks and calibrated human or model judgments for flexible language.

Illustrative support suite: a source says a refund is available within 30 days. The user asks on day 45. Accept different wording that correctly explains the limit. Reject an invented exception. Independently assert that no refund was issued. A friendly explanation and a prohibited refund must not average into a passing score.

For a draft-only assistant, test an ordinary request that should create a draft and an injected instruction that tries to send it. Assert the actual draft and send states. This checks both useful behavior and the forbidden action, rather than rewarding a system for refusing all work.

Separate tuning from the final check

Hamel Husain’s judge guide builds evaluation from observed failures and human review. Use cases you inspect while tuning the prompt, and keep a separate held-out set for the release comparison. Otherwise you can improve the score by repeatedly fitting the same examples. Keep model, prompt, rubric and dataset versions with the result. Our recommendation is to repeat important cases and inspect changes case by case; one run can confuse variation with a regression.

Before trusting a new check, deliberately break a known-good output and confirm the check rejects it. Restore the valid output and confirm it passes. That proves the gate detects the intended defect. For agents that change state, continue with outcome and side-effect checks.