# What base classifiers to use with bagging?

I am working on a specific implementation of bootstrap aggregating (bagging). I want to see how well this bagging works for different base classifiers. But so far, the decision tree seems to be the only option that I'm running into. I tried Naive Bayes but bagging seems to make things worse for some data sets. Can you suggest some more algorithms that could be used as base learner for bagging?

Basically you can bag any base learner. Following diversification idea, they should tend to have low bias (= overfit!) and, consequently, high variance.

Examples would include:

• deep decision trees
• $k$-nearest neighbour with low $k$
• richly parametrized linear models (e.g. with splines, pairwise interactions etc.)
• neural nets with enough complexity

Not very suitable from this perspective are

• tree stumps
• too simple linear models

Bagging build new models using the same classifier on variants of the data set. If the classifier is very stable, the models will have a lot of agreement and you won't gain too much from the bagging. The less stable the classifier, the more likely you will gain.

In the original paper about bagging, Breiman refers to this point.

Unstability was studied in Breiman [1994] where it was pointed out that neural nets, classification and regression trees, and subset selection in linear regression were unstable,while k-nearest neighbor methods were stable.

Breiman [1994] is "Breiman,L.(1994)Heuristics of instability in model selection,Technical Report, Statistics Department, University of California at Berkeley."

So decision trees are indeed good candidate classifier for bagging but so are the classifiers listed above in an answer by Michael M and other unstable classifiers. Not that stability is also a function of the data itself.