A payments company might run three different fraud models: one good at spotting stolen cards, one at account takeovers and one simple rule-based score. Each misses cases the others catch. Taking a vote, or averaging their risk scores, can give a combined answer that is wrong less often than an individual model. Diversity helps, but it does not guarantee beating the strongest member: compare the ensemble with each model on held-out data. Three identical models making identical predictions add cost without adding accuracy.
The main families differ in how they create that variety. Bootstrap aggregating (bagging) trains copies of one model on different resamples of the data; a random forest is the best-known example. Boosting adds models one at a time, each correcting what the group so far gets wrong. Voting combines different kinds of model directly, and stacking trains a final model to learn how much to trust each member.
Ensembles cost more to train, serve and monitor, and they can be slower to answer and harder to explain. They don’t fix bad labels or leaked test data. In large language model products, mixture of agents applies a similar combine-several-answers idea to LLM outputs. A mixture of experts is different: it is one model whose internal parts are routed per input, not a group of separately trained models.
Sources
- scikit-learn: Ensembles: gradient boosting, random forests, bagging, voting, stacking — Documents the main ensemble families, including bagging, random forests, gradient boosting, voting and stacked generalization, where a final estimator learns from the base models' predictions; checked October 24, 2026.