Ensembles of Ensembles?

Question

I like the idea of ensemble learners, especially Bagging, but I always wondered as why they are not the most powerful learners since they have a clean motivation. I don't have the answer to that question but I had another idea.

Normally in Bagging people use the same classifier for learning. So they divide the dataset into slices and for each slice they train a classifier of the same type (e.g. logistic regression) and then they use voting.

But my question is why not to use ensembles of ensembles? Why not to create a bagging classifier of logistic regression, a bagging classifier of SVM, a bagging classifier of ANN, a bagging classifier of random-forest, and then use voting. So each classifier is an ensemble and then all the ensembles become an ensemble. Then use voting again.

Has anyone tried this before? Papers? ... etc? There must be!

Answer 1

Although not a method for ensembles for ensembles, I see many parallels with:

Caruana, Rich, Alexandru Niculescu-Mizil, Geoff Crew, and Alex Ksikes. "Ensemble selection from libraries of models." In Proceedings of the twenty-first international conference on Machine learning, p. 18. ACM, 2004.

Available without a paywall here: http://www.cs.cornell.edu/~caruana/ctp/ct.papers/caruana.icml04.icdm06long.pdf

The idea is to construct an ensemble from a diverse range of classifiers such as SVMs, ANNs, KNN and decision trees. Furthermore, rather than optimizing the parameters of each of the individual classifiers, simply include one classifier for each parameter value in the library from which the ensemble is constructed.

The resulting ensemble is called a heterogeneous ensemble, contrasted with more common homogeneous ensembles like random forests where the base learners are all of the same type.

Such heterogeneous ensembles have been shown to achieve state-of-the-art classification performance in credit risk. See Lessman et al (2013) "Benchmarking state-of-the-art classification algorithms for credit scoring: A ten-year update" available here: http://www.business-school.ed.ac.uk/waf/crc_archive/2013/42.pdf

Answer 2

For pointers to the literature, see the nice answer of @M. Berk. I have just a small comment, which might explain why such methods are not ubiquituous and probably won't be.

From the first I would be unsure whether the result pays the effort. The standard argument for Bagging shows that the variance decreases, as long as the results are uncorrelated. More detailed (citing Hastie et. al., chapter 15), if you have $B$ i.i.d. random variables each with variance $\sigma$ and pair-wise correlation $\rho$, the variance of the average is

$$\rho \sigma^2 + \frac{1-\rho}{B} \sigma^2.$$

If you now have a well tuned random forest ensemble, I guess any of those universal methods you mentioned will probably have a large correlation with the random forest prediction, and as a consequence you will hardly get an improvement. The same will probably hold for any other well tuned ensemble (let it be ANNs, SVMs, etc.).

Moreover, putting more and more models in your ensemble can also lead to overfitting (if the ensemble is not properly regularized).