The classic picture from the bias-variance tradeoff says test error falls as a model gets more flexible, bottoms out, then rises as the model starts overfitting. In 2018, Mikhail Belkin and colleagues showed that the curve can keep going. Error peaks around the “interpolation threshold,” the size at which the model can just barely fit every training example exactly, and then falls again as the model grows well past it.
A team comparing three network sizes on the same task might see the small one underfit, the medium one score worst on validation data while fitting training data almost perfectly, and the large one beat both. A 2019 study by Preetum Nakkiran and colleagues at Harvard and OpenAI found the same shape in deep networks, not only as models get bigger but also as they train for more epochs. Near the peak, adding training data can even make results worse for a while. Noisy or mislabeled data makes the peak more pronounced.
Double descent helps explain why very large models can generalize well despite having enough capacity to memorize their training data. It is not a promise that bigger always wins on your data, and the location of the peak depends on model, data size and noise. Treat it as a reason not to stop at the first “sweet spot,” and keep measuring on held-out data. For how loss changes with scale in large language models, see scaling laws.
Sources
- Belkin, Hsu, Ma and Mandal: Reconciling modern machine learning practice and the bias-variance trade-off — Names the double descent curve and shows that increasing capacity beyond the point of interpolation can improve performance across a range of models and datasets; checked October 24, 2026.
- Nakkiran et al.: Deep Double Descent: Where Bigger Models and More Data Hurt — Reports double descent in modern deep learning as a function of model size and of training epochs, including settings where more data hurts; checked October 24, 2026.