I am working on a specific implementation of bootstrap aggregating (bagging). I want to see how well this bagging works for different base classifiers. But so far, the decision tree seems to be the only option that I'm running into. I tried Naive Bayes but bagging seems to make things worse for some data sets. Can you suggest some more algorithms that could be used as base learner for bagging?


3 Answers 3


Basically you can bag any base learner. Following diversification idea, they should tend to have low bias (= overfit!) and, consequently, high variance.

Examples would include:

  • deep decision trees
  • $k$-nearest neighbour with low $k$
  • richly parametrized linear models (e.g. with splines, pairwise interactions etc.)
  • neural nets with enough complexity

Not very suitable from this perspective are

  • tree stumps
  • too simple linear models

Bagging build new models using the same classifier on variants of the data set. If the classifier is very stable, the models will have a lot of agreement and you won't gain too much from the bagging. The less stable the classifier, the more likely you will gain.

In the original paper about bagging, Breiman refers to this point.

Unstability was studied in Breiman [1994] where it was pointed out that neural nets, classification and regression trees, and subset selection in linear regression were unstable,while k-nearest neighbor methods were stable.

Breiman [1994] is "Breiman,L.(1994)Heuristics of instability in model selection,Technical Report, Statistics Department, University of California at Berkeley."

So decision trees are indeed good candidate classifier for bagging but so are the classifiers listed above in an answer by Michael M and other unstable classifiers. Not that stability is also a function of the data itself.

For more information see here.



1) wavelets,

2) other basis functions which are non-zero only on a relatively small subspace of the data.

What are your research questions and data? What are you trying to do?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.