I like the idea of ensemble learners, especially Bagging, but I always wondered as why they are not the most powerful learners since they have a clean motivation. I don't have the answer to that question but I had another idea.
Normally in Bagging people use the same classifier for learning. So they divide the dataset into slices and for each slice they train a classifier of the same type (e.g. logistic regression) and then they use voting.
But my question is why not to use ensembles of ensembles? Why not to create a bagging classifier of logistic regression, a bagging classifier of SVM, a bagging classifier of ANN, a bagging classifier of random-forest, and then use voting. So each classifier is an ensemble and then all the ensembles become an ensemble. Then use voting again.
Has anyone tried this before? Papers? ... etc? There must be!