What base classifiers to use with bagging? I am working on a specific implementation of bootstrap aggregating (bagging). I want to see how well this bagging works for different base classifiers. But so far, the decision tree seems to be the only option that I'm running into. I tried Naive Bayes but bagging seems to make things worse for some data sets. Can you suggest some more algorithms that could be used as base learner for bagging?
 A: Basically you can bag any base learner. Following diversification idea, they should tend to have low bias (= overfit!) and, consequently, high variance.
Examples would include:


*

*deep decision trees

*$k$-nearest neighbour with low $k$

*richly parametrized linear models (e.g. with splines, pairwise interactions etc.)

*neural nets with enough complexity


Not very suitable from this perspective are


*

*tree stumps

*too simple linear models

A: Bagging build new models using the same classifier on variants of the data set.
If the classifier is very stable, the models will have a lot of agreement and you won't gain too much from the bagging.
The less stable the classifier, the more likely you will gain.
In the original paper about bagging, Breiman refers to this point.

Unstability was studied in Breiman [1994] where it was pointed
out that neural nets, classification and regression trees, and subset selection
in linear regression were unstable,while k-nearest neighbor methods were
stable.

Breiman [1994] is "Breiman,L.(1994)Heuristics of instability in model selection,Technical
Report, Statistics Department, University of California at Berkeley."
So decision trees are indeed good candidate classifier for bagging but so are the classifiers listed above in an answer by Michael M and other unstable classifiers.
Not that stability is also a function of the data itself.
For more information see here.
A: Potentially,
1) wavelets, 
2) other basis functions which are non-zero only on a relatively small subspace of the data. 
What are your research questions and data? What are you trying to do?
