A lender predicting loan defaults could train 200 decision trees. Each tree gets its own “bootstrap sample”: rows drawn at random from the training data with replacement, so some loans appear twice and others are left out. To score a new application, every tree votes and the majority wins. Because each tree saw slightly different data, their individual quirks partly cancel out.
That is why bagging helps most with unstable models, ones whose predictions change a lot when the training data changes a little, which is high variance in bias-variance terms. Rows a tree never saw also give a free check: score each loan using only the trees that left it out, an “out-of-bag” estimate. A random forest is bagging plus one more source of randomness: each tree considers only a random subset of features at each split.
Bagging is one of several ensemble methods. Unlike boosting, where each new model focuses on the previous models’ mistakes, bagged models train independently and can run in parallel. Bagging doesn’t fix a model that is systematically wrong, and the out-of-bag estimate is only as good as the original sample is representative.
Sources
- Breiman: Bagging Predictors (Berkeley technical report) — Introduces bagging: models trained on bootstrap replicates of the learning set, averaged for numbers and combined by plurality vote for classes, with gains for unstable procedures such as classification and regression trees; checked October 24, 2026.
- scikit-learn: Bagging meta-estimator — Describes bagging as building several instances of an estimator on random subsets of the training set, drawn with replacement, to reduce the variance of a complex base model; checked October 24, 2026.