In detail
Give the generator something to work from
Synthetic test data consists of constructed cases rather than cases collected directly from users. It can help you explore a behavior before you have many examples of it. The hard part is deciding whether the generated case represents a real task and whether its expected answer is correct.
In episode 57 at 29:26, published April 2026, Alex Ratner argues for expert involvement in generating, curating and reviewing data. At 28:05, he estimates review and quality control take roughly 40 to 50% of the time Snorkel spends creating a data point or environment, at what he calls its own high quality bar. Asking a model to invent questions and certify its own answers leaves you without an independent check. Start with approved documents, reviewed examples or a failure you have already observed.
Generate situations, not just paraphrases
Illustrative example: a workplace equipment policy allows replacement after three years, with a separate approval process for accidental damage. Give the generator that policy and ask for cases that exercise different decisions: an ordinary replacement, an early request, a damaged device and a request with no purchase date.
For each case, ask it to propose a question, the relevant policy passage, an expected response and a reason the case belongs in the test set. A reviewer should verify all four. When the purchase date is missing, the expected response may be a clarification question; the generator should not invent a date to make the answer easy.
Ragas’s test-generation guide distinguishes questions answerable from one piece of information from questions that combine several. That is a useful starting structure for a retrieval test set. It does not establish that a generated answer is correct for your policy.
Review diversity and keep a separate check
In episode 43 at 28:40, published November 2025, Maxime Labonne points out that many generated variations can still come from the same process and have limited diversity. Inspect the decisions your cases exercise. Twenty rewrites of an ordinary replacement request do not cover the damaged-device exception.
Make a small coverage table before generating more rows. Include the intended decision, available evidence, missing information and expected next action. Add real failures as they become available, with appropriate data handling. Compare whether generated cases reflect the wording, ambiguity and conversation length that actual users bring.
Amplitude’s chief engineering officer, Wade Chambers, described a two-sided comparison in episode 24 at 13:36, published in May 2025. Amplitude watches what its internal Ask Amplitude agent is asked and how it responds, to catch hallucinations (13:07). For evaluation, he says, the team generates synthetic results and compares outputs against them as well as against previous norms, which gives it a historical view and a generated one. He doesn’t say how the synthetic results are built. The episode aired in season two, which Galileo sponsored while Conor worked there.
Keep tests used for final evaluation apart from the cases used to tune prompts or train models. If a case becomes a development example, treat it as development material and preserve an untouched check elsewhere. Group near-duplicates so a reworded answer key does not cross that boundary.
Where it falls short
Synthetic data can help test a specified scenario, but the number of generated failures does not tell you how often that scenario occurs in the world. A deliberately difficult test set and a representative sample answer different questions. Label them accordingly.
Also inspect the scoring process. A correct question with an incorrect expected answer can punish the system for being right. When reviewers disagree, resolve the source interpretation or mark the case unresolved before using it as a gate. Testing nondeterministic AI covers what to do once you have trustworthy cases.
Disclosure: Episode 57 was sponsored by Galileo, the Season 3 presenting sponsor (thanks at 1:36). Episode 24 aired in season two, which Galileo sponsored while Conor worked there; its show notes carry a Galileo block.
Hear it from the guest
“But, actually, it's the same data generation process. And because it's the same data generation process, it will limit the diversity.”
“You have to have sophisticated ways of having kind of human or expert in the loop, but AI-accelerated processes for all of the core operations of generating data, generating environments, curating those, reviewing those, et cetera.”
“We'll go through and generate synthetic results and then compare and contrast to that as well as previous norms”
Quotes lightly edited to remove filler words.
Go deeper
- Testset Generation for RAG Shows how document relationships and different query types can guide test generation.
- Discovering Language Model Behaviors with Model-Written Evaluations Compare the paper's model-generated evaluations and human quality checks. Use its reported failure cases and hybrid generation process to plan review of your own test cases.
From the conversation
This explainer is drawn from these episodes — each carries its full transcript.