A teacher who hands out the final exam as homework learns nothing from the exam scores. Machine learning has the same problem, so teams split their dataset before training. A common split is something like 80% training, 10% validation and 10% test, though the right sizes depend on how much data exists.
Each slice has one job. The training set is what the model fits its parameters to. The validation set (also called a dev set) is what people check during development: which version is better, which hyperparameter setting wins, when to stop training before overfitting sets in. The test set is opened at the end to estimate how the chosen model will do on genuinely new data. Google’s course warns that validation and test sets “wear out”: every time a team peeks at test results and adjusts, the test set becomes a little more like training data and the score becomes a little less trustworthy. Cross-validation is a way to get more out of limited data during the validation stage.
The split only works if information can’t cross it. Duplicates, near-duplicates or the same customer appearing in both training and test is a form of data leakage, and for time-ordered data the test set should come from later dates. The same idea carries into LLM evaluation: an eval set kept out of prompts and fine-tuning data, and public benchmarks that may have leaked into pre-training data (benchmark contamination).
In episode 34 at 41:24, Corti CEO Andreas Cleve says building clinical AI takes many datasets and “a lot of holdouts,” then checks the output inside the doctor’s workflow, where a doctor’s sign-off acts as a further label. That is his account of Corti’s process. The episode was sponsored by Galileo, where Conor worked at the time (33:48).
Hear it from the guest
“So we do we do need a lot of data sets. We need a lot of holdouts”
Quotes lightly edited to remove filler words.
Sources
- Google Machine Learning Crash Course: Dividing the original dataset — Recommends splitting data into training, validation and test sets, using the validation set during development and the test set for final evaluation, and warns that both can wear out with repeated use; checked October 24, 2026.
- Google Machine Learning Glossary: validation set and test set — Says each example should belong to only one subset and that the test set gives a less biased measure because it is only indirectly tied to training; checked October 24, 2026.
