# Resources for learning how to implement ensemble methods

I understand theoretically (sort of) how they would work, but am not sure how to go about actually making use an ensemble method (such as voting, weighted mixtures, etc.).

• What are good resources for implementing ensemble methods?
• Are there any particular resources regarding implementation in Python?

EDIT:

To clear up some based on the discussion on the comments, I'm not looking for ensemble algorithms such as randomForest, etc. Instead, I'm wondering how can you combine different classifications from different algorithms.

For example, say someone uses logistic regression, SVM, and some other methods to predict the class of a certain observation. What is the best way to go about capturing the best estimate of the class based upon these predictions?

-

A good place to start is to get an overview of ensemble learning. Especially you'll want to look at boosting and bagging. Another method was that used by "The Ensemble" team in the Netflix Prize, is called either "blending" or feature stacking.

Then, just find some libraries that implement those and work from there. A quick googling turned up scikit and orange, both of which should have bagging and boosting (and they're both Python).

If beyond just using ensemble methods, you'd like to learn a bit of the theory, then I think this paper would be a good jumping off point (follow the references for the parts you're interested in).

Cheers.

-
(+1) woa, great references here :O ! –  steffen Sep 19 '12 at 6:49
Thanks. Just trying to contribute something about one of the few topics I know anything about. –  Stumpy Joe Pete Sep 19 '12 at 12:55

'Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions', Seni and Elder -- Excellent reference on practical ensemble theory and implementation, but accompanying code is R based.

'Machine Learning: An Algorithmic Perspective,' S. Marsland -- Excellent Python based practical text, but not as dedicated to pure ensemble concepts as the first reference.

-

Salford Systems has a software package called Random Forests that implements this for classification and regression tree ensembles. I don't have any free R packages to offer. I imagine they have a users manual that will explain their implementation. By analogy you could probably figure out how to do it for other ensemble methods.

-
There are many great R packages for ensembles of trees: e.g. randomForest (classic algorithm), party::cforest (random forest using conditional inference trees), gbm (gradient boosting of trees) to name a few. I read the OP as wanting to implement classifier/regression agnostic ensembles. The simplest procedure is of course to average predictions. –  B_Miner Jul 20 '12 at 17:03
@B_Miner It is nice to know that there are implementations available in R. Maybe somebody could explain to me why a specific implementation in Python is desirable (please excuse my ignorance about R). I read the the OP to want to know sources that describe how to implement the ensemble methods. The Salford package was one that I was aware of that might have some documentation. –  Michael Chernick Jul 20 '12 at 17:20
Thank you both. I've edited the OP to clear up some. –  user1074057 Jul 20 '12 at 21:06
While based on the Freund and Schapire paper boosting works in general as far as I know the best results have come using tree classifiers. –  Michael Chernick Jul 20 '12 at 21:24
I personally get really good results by simple averaging of probabilities - but my domain is more interested in probabilities than picking a class label. –  B_Miner Jul 21 '12 at 0:33
show 1 more comment

I found this tutorial which was extremely helpful. It doesn't answer all the pieces but I think it is a great start to the discussion: http://vikparuchuri.com/blog/intro-to-ensemble-learning-in-r/

-