Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

It seems to me that ensemble learning WILL always give better predictive performance than with just a single learning hypothesis.

So, why don't we use them all the time?

My guess is because of perhaps, computational limitations? (even then, we use weak predictors, so I don't know).

share|improve this question
Because ensemble learning doesn't always give better performance. Both bagging and boosting work in some cases but can severely degrade performance in others. – Marc Claesen Jul 29 '14 at 14:35
up vote 5 down vote accepted

In general it is not true that it will always perform better. There are several ensemble methods, each with its own advantages/weaknesses. Which one to use and then depends on the problem at hand.

For example, if you have models with high variance (they over-fit your data), then you are likely to benefit from using bagging. If you have biased models, it is better to combine use them with Boosting. There are also different strategies to form ensembles. The topic is just too wide to cover it in one answer.

But my point is: if you use the wrong ensemble method for your setting, you are not going to do better. For example, using Bagging with a biased model is not going to help.

Also, if you need to work in a probabilistic setting, ensemble methods may not work either. It is known that Boosting (in its most popular forms like AdaBoost) delivers poor probability estimates. That is, if you would like to have a model that allows you to reason about your data, not only classification, you might be better off with a graphical model.

share|improve this answer
A decision stump is biased, but they have been used successfully with bagging. – sponge_knight Jul 29 '14 at 13:57
yes, but the ensemble is still biased. What if bias is really an issue?. Bagging won't help to fix it. Could you add a reference to that case you mention? – jpmuc Jul 29 '14 at 14:09

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.