Name what should be new
A support classifier might need to handle another message from an existing customer, a new customer or next month’s workload. Those are different evaluation questions. Google’s overfitting explanation contrasts matching training examples with predicting new data. Define which kind of new data matters before interpreting a held-out score.
In episode 49 at 15:04, Fergal Reid warns that teaching a model the kind of task in a benchmark can still leave it failing to generalize. That is the case for testing on your own tasks.
Match the split to the question
Fictional support dataset: several customers each have multiple tickets, some nearly identical. Consider three ways of holding out records.
| Split by | Keep apart | Question |
|---|---|---|
| Ticket | Individual records | Another ticket from sampled customers? |
| Customer | All records per customer | Customers absent from training? |
| Time | Earlier and later records | A later period in this stream? |
A random ticket split can put one customer’s near-duplicate messages on both sides. That does not answer the new-customer question. Holding out complete customers addresses that overlap; a later-period test addresses a different boundary. If the intended use requires both new customers and future periods, design for both constraints.
scikit-learn’s cross-validation guide explains grouped and time-aware evaluation. Preserve relevant ordering, and use only information that would have been available when the prediction was made. A date boundary does not rescue features computed with future information.
Protect the final comparison and its limits
Training fits the model; validation or cross-validation helps guide development; a separate test evaluates the chosen system. Reusing final-test feedback for choices weakens that separation even when only the prompt changes. Keep the selection history visible and distinguish this risk from benchmark contamination.
A training-to-test gap can indicate overfitting, but also warrants checking data differences and leakage. A policy change, a language the test set never covered or next quarter’s traffic can each break a model that passed its holdout. Report the task, split unit, time period, available inputs and uncertainty, then check outcomes as the workload changes. See how to test an AI system for the broader evaluation workflow.
Hear it from the guest
“Get good at teaching the model how to do the kind of task that's in the benchmark and then suffer generalization failure.”
Quotes lightly edited to remove filler words.
Sources
- Google: Overfitting — Text explanation contrasts fitting training examples with predicting new data; checked November 2, 2026.
- scikit-learn: Cross-validation — Opening evaluation guidance and grouped/time-series sections explain selection separation and why related or ordered samples need an appropriate split.
Go deeper
- scikit-learn: Common pitfalls docs
See how preprocessing leakage and inconsistent transformations undermine unseen-data evaluation.