A retailer builds a model to predict which orders will be returned. Accuracy in testing is spectacular. In production it is barely better than guessing. The cause: one input column was “refund processed,” which is only filled in after a return happens. The model learned to read the answer off that column. Google’s glossary gives a similar example of a “spoke to customer agent” flag that only gets set after a purchase.
Leaks come in a few common shapes. Target leakage is a feature that is a stand-in for the answer. Train-test contamination is the same or near-duplicate examples on both sides of the train/test split. Preprocessing leakage, highlighted by scikit-learn, comes from computing averages or feature scaling over the full dataset before splitting. Temporal leakage is training on data from after the moment you’re predicting, a constant risk in time-series forecasting.
This is not a beginner-only mistake. Kapoor and Narayanan found leakage errors in 329 published papers across 17 fields. Prevention is mostly discipline: split first, fit every preprocessing step on training data only, check that each feature would exist at prediction time, and treat a suspiciously good score as a bug until proven otherwise. In the LLM world the closest cousin is benchmark contamination, where test questions end up in pre-training data. The security sense of “data leak,” exposing private information, is a different problem; see token leakage for one example.
Sources
- scikit-learn: Common pitfalls and recommended practices, data leakage — Defines data leakage as using information that would not be available at prediction time, which gives overly optimistic estimates; recommends splitting before any preprocessing and never fitting on test data; checked October 24, 2026.
- Kapoor and Narayanan: Leakage and the Reproducibility Crisis in ML-based Science — Finds leakage errors in 17 research fields affecting 329 papers, sometimes producing wildly overoptimistic conclusions, and proposes a taxonomy of eight leakage types; checked October 24, 2026.
- Google Machine Learning Glossary: label leakage — Describes label leakage as a feature that is a proxy for the label, with a customer-agent flag assigned only after a purchase as the example; checked October 24, 2026.