It seems to me that ensemble learning WILL always give better predictive performance than with just a single learning hypothesis.

So, why don't we use them all the time?

My guess is because of perhaps, computational limitations? (even then, we use weak predictors, so I don't know).

  • 3
    $\begingroup$ Because ensemble learning doesn't always give better performance. Both bagging and boosting work in some cases but can severely degrade performance in others. $\endgroup$ Jul 29, 2014 at 14:35

2 Answers 2


In general it is not true that it will always perform better. There are several ensemble methods, each with its own advantages/weaknesses. Which one to use and then depends on the problem at hand.

For example, if you have models with high variance (they over-fit your data), then you are likely to benefit from using bagging. If you have biased models, it is better to combine use them with Boosting. There are also different strategies to form ensembles. The topic is just too wide to cover it in one answer.

But my point is: if you use the wrong ensemble method for your setting, you are not going to do better. For example, using Bagging with a biased model is not going to help.

Also, if you need to work in a probabilistic setting, ensemble methods may not work either. It is known that Boosting (in its most popular forms like AdaBoost) delivers poor probability estimates. That is, if you would like to have a model that allows you to reason about your data, not only classification, you might be better off with a graphical model.

  • 1
    $\begingroup$ A decision stump is biased, but they have been used successfully with bagging. $\endgroup$
    – user46925
    Jul 29, 2014 at 13:57
  • $\begingroup$ yes, but the ensemble is still biased. What if bias is really an issue?. Bagging won't help to fix it. Could you add a reference to that case you mention? $\endgroup$
    – jpmuc
    Jul 29, 2014 at 14:09

You should ! If you refer to ensemble of models (also called blending, staging, stacking). Unfortunately, the sense behind ensemble is quite vague. Models themselves based on ensemble (Random Forest...) may not performed better than others (Linear Regression, Neural networks).

In his article: Stacked generalization (1992) David H. Wolpert stated:

The conclusion is that for almost any real-world generalization problem one should use some version of stacked generalization to minimize the generalization error rate.

And this remains true until today.

However, these model are harder to tune and ship than a single model. They also have a longer prediction time, since you need the predictions of possibly hundreds of models. And, even though they have better predicitions, the increment in accuracy may not be worth the hassle.

After seeing various questions about ensemble learning, I wrote why does model staging work exploring the reasons of their success.

  • $\begingroup$ Do you mean "model stacking" instead of "model staging"? $\endgroup$
    – Sycorax
    Jun 9, 2022 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.