Ensemble methodology's main aim is to somehow aggregate or summarize estimates from multiple models. In some cases this is aggregating different bootstrap estimates or Monte Carlo estimates, but others involve combining the estimates from starkly different models. When you look up "ensemble methods", especially in ESL, they mention boosting, bagging, bumping, random forests, and the EM algorithm. Question is: is this all of them? Or more generally, are all so-called ensemble methods somehow contained within these classes? Or is there a more general term that I'm missing?

The list so far:

  • Boosting
  • Bagging
  • Bumping
  • Random forests
  • EM algorithm

Can you add to this?


Ensemble methods arise when there is uncertainty about a particular model structure. The goal is to draw inference from a set, or ensemble, of candidate models. In a general sense, ensemble methods are a form of model averaging. The model may be a classification tree (random forests) or a regression model, or whatever. Several of the terms you've listed are simply alternative ways to take the average across models. Boosting is a term where models are sampled randomly, whereas other methods such as bagging and bumping more strongly emphasize training where the averaging is adaptive based on how well a particular model is doing. So I believe the more general class of methods you're interested in is 'model averaging' where the ensemble represents the set of candidate models to be averaged over.

  • $\begingroup$ Yes, I'm thinking of chapter 8 of ESL. The question is: how do you take the average? Straight sample mean of the estimates? Probably we can do better. A weighted integral seems like a likely choice, but how do we choose the weights? $\endgroup$ – owensmartin Jun 19 '13 at 18:15
  • $\begingroup$ @owensmartin As I mentioned in the answer, there are many ways to take this kind of model ensemble average - the various methods you mentioned are all approaches, some more sophisticated than others, and all with their own pros and cons. Do you have a particular example? $\endgroup$ – gregory_britten Jun 19 '13 at 19:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.