Skip to content

AI Glossary · Concepts

What is Generalization?

Generalization

In machine learning, generalization is a model's ability to perform well on examples beyond those used to fit it. That ability is measured for a particular task and data distribution, not for every possible future input.

· Updated · Chain of Thought

AI Evaluation & ReliabilityModel Training

  • 1 explainer
  • 1 episode
  • 1 guest clip
  • 2 sources

Name what should be new

A support classifier might need to handle another message from an existing customer, a new customer or next month’s workload. Those are different evaluation questions. Google’s overfitting explanation contrasts matching training examples with predicting new data. Define which kind of new data matters before interpreting a held-out score.

In episode 49 at 15:04, Fergal Reid warns that teaching a model the kind of task in a benchmark can still leave it failing to generalize. That is the case for testing on your own tasks.

Match the split to the question

Fictional support dataset: several customers each have multiple tickets, some nearly identical. Consider three ways of holding out records.

Split byKeep apartQuestion
TicketIndividual recordsAnother ticket from sampled customers?
CustomerAll records per customerCustomers absent from training?
TimeEarlier and later recordsA later period in this stream?

A random ticket split can put one customer’s near-duplicate messages on both sides. That does not answer the new-customer question. Holding out complete customers addresses that overlap; a later-period test addresses a different boundary. If the intended use requires both new customers and future periods, design for both constraints.

scikit-learn’s cross-validation guide explains grouped and time-aware evaluation. Preserve relevant ordering, and use only information that would have been available when the prediction was made. A date boundary does not rescue features computed with future information.

Protect the final comparison and its limits

Training fits the model; validation or cross-validation helps guide development; a separate test evaluates the chosen system. Reusing final-test feedback for choices weakens that separation even when only the prompt changes. Keep the selection history visible and distinguish this risk from benchmark contamination.

A training-to-test gap can indicate overfitting, but also warrants checking data differences and leakage. A policy change, a language the test set never covered or next quarter’s traffic can each break a model that passed its holdout. Report the task, split unit, time period, available inputs and uncertainty, then check outcomes as the workload changes. See how to test an AI system for the broader evaluation workflow.

Hear it from the guest

“Get good at teaching the model how to do the kind of task that's in the benchmark and then suffer generalization failure.”
Watch on YouTube

Quotes lightly edited to remove filler words.

Sources

  • Google: Overfitting — Text explanation contrasts fitting training examples with predicting new data; checked November 2, 2026.
  • scikit-learn: Cross-validation — Opening evaluation guidance and grouped/time-series sections explain selection separation and why related or ordered samples need an appropriate split.

Go deeper

From the conversation