Suppose you predict house prices. A straight-line model that uses only floor area misses the fact that the same space costs far more in some neighborhoods; retrain it on different sales and it makes the same systematic mistakes. That is high bias, or underfitting. A very deep decision tree can fit last year’s sales almost perfectly, but retrained on a different sample of sales it gives noticeably different prices for the same house. That is high variance, the root of overfitting.
For squared-error prediction, statisticians split the expected error on new data into three parts: bias squared, variance and noise that no model can remove. Making a model more flexible usually shrinks the first part and grows the second, so the best test error often comes somewhere in between. Tools such as regularization, more training data and bootstrap aggregating mainly attack variance; a richer model or better features attack bias.
The “bias” here is a statistical term, not the fairness sense of bias in AI products. The tradeoff is also a guide rather than a law: very large modern models can show double descent, where test error falls again after the point where the model fits its training data exactly. Measure on held-out data instead of assuming where the sweet spot is.
Sources
- scikit-learn: Validation curves — States that an estimator's generalization error can be decomposed into bias, variance and noise, and shows validation curves for underfitting and overfitting; checked October 24, 2026.
- scikit-learn: Single estimator versus bagging, bias-variance decomposition — Works through the squared-error decomposition into bias squared, variance and noise and shows bagging lowering variance; checked October 24, 2026.
- James, Witten, Hastie and Tibshirani: An Introduction to Statistical Learning, Chapter 2 — Section 2.2.2 defines variance as how much the fitted model would change with a different training set and shows that expected test error cannot fall below the irreducible error; checked October 24, 2026.