# Resources for learning how to implement ensemble methods

I understand theoretically (sort of) how they would work, but am not sure how to go about actually making use an ensemble method (such as voting, weighted mixtures, etc.).

• What are good resources for implementing ensemble methods?
• Are there any particular resources regarding implementation in Python?

EDIT:

To clear up some based on the discussion on the comments, I'm not looking for ensemble algorithms such as randomForest, etc. Instead, I'm wondering how can you combine different classifications from different algorithms.

For example, say someone uses logistic regression, SVM, and some other methods to predict the class of a certain observation. What is the best way to go about capturing the best estimate of the class based upon these predictions?

A good place to start is to get an overview of ensemble learning. Especially you'll want to look at boosting and bagging. Another method was that used by "The Ensemble" team in the Netflix Prize, is called either "blending" or feature stacking.

Then, just find some libraries that implement those and work from there. A quick googling turned up scikit and orange, both of which should have bagging and boosting (and they're both Python).

If beyond just using ensemble methods, you'd like to learn a bit of the theory, then I think this paper would be a good jumping off point (follow the references for the parts you're interested in).

Cheers.

• (+1) woa, great references here :O ! Sep 19 '12 at 6:49
• Thanks. Just trying to contribute something about one of the few topics I know anything about. Sep 19 '12 at 12:55

'Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions', Seni and Elder -- Excellent reference on practical ensemble theory and implementation, but accompanying code is R based.

'Machine Learning: An Algorithmic Perspective,' S. Marsland -- Excellent Python based practical text, but not as dedicated to pure ensemble concepts as the first reference.

Stumpy Joe Pete's response was perfect, but since you mentioned about a Python implementation, I wanted to mention the brew project from Universidade Federal de Pernambuco.

https://github.com/viisar/brew

from brew.base import Ensemble
from brew.base import EnsembleClassifier
from brew.combination import import Combiner

clfs = your_list_of_classifiers # [clf1, clf2]
ens = Ensemble(classifiers = clfs)

# the rules can be 'majority_vote', 'max', 'min', 'mean' or 'median'
comb = Combiner(rule='majority_vote')

# now create your ensemble classifier
ensemble_clf = EnsembleClassifier(ensemble=ens, combiner=comb)
ensemble_clf.predict(X)


At this point, they have ensemble generation, combination, pruning and dynamic selection.

Limitations: classification only; no stacking in current public version; not much documentation.

Salford Systems has a software package called Random Forests that implements this for classification and regression tree ensembles. I don't have any free R packages to offer. I imagine they have a users manual that will explain their implementation. By analogy you could probably figure out how to do it for other ensemble methods.

• There are many great R packages for ensembles of trees: e.g. randomForest (classic algorithm), party::cforest (random forest using conditional inference trees), gbm (gradient boosting of trees) to name a few. I read the OP as wanting to implement classifier/regression agnostic ensembles. The simplest procedure is of course to average predictions. Jul 20 '12 at 17:03
• @B_Miner It is nice to know that there are implementations available in R. Maybe somebody could explain to me why a specific implementation in Python is desirable (please excuse my ignorance about R). I read the the OP to want to know sources that describe how to implement the ensemble methods. The Salford package was one that I was aware of that might have some documentation. Jul 20 '12 at 17:20
• While based on the Freund and Schapire paper boosting works in general as far as I know the best results have come using tree classifiers. Jul 20 '12 at 21:24
• I personally get really good results by simple averaging of probabilities - but my domain is more interested in probabilities than picking a class label. Jul 21 '12 at 0:33
• @MichaelChernick If you're doing really intense predictive work (like... a Kaggle competition), you're not going to pick either boosting or random forests. You're going to want to aggregate as many possible models as will help you (which is generally a lot more than one). So, in that context, other ensemble methods are going to be important, even if random forests are way awesomer than anything else. Sep 19 '12 at 2:44

I found this tutorial which was extremely helpful. It doesn't answer all the pieces but I think it is a great start to the discussion: http://vikparuchuri.com/blog/intro-to-ensemble-learning-in-r/

Scikit-learn ensembling guide provides bagging and boosting meta-classifiers and regressors. In addition, mlxtend library provides implementations of stacking meta-classifiers and regressors.