14
$\begingroup$

I understand theoretically (sort of) how they would work, but am not sure how to go about actually making use an ensemble method (such as voting, weighted mixtures, etc.).

  • What are good resources for implementing ensemble methods?
  • Are there any particular resources regarding implementation in Python?

EDIT:

To clear up some based on the discussion on the comments, I'm not looking for ensemble algorithms such as randomForest, etc. Instead, I'm wondering how can you combine different classifications from different algorithms.

For example, say someone uses logistic regression, SVM, and some other methods to predict the class of a certain observation. What is the best way to go about capturing the best estimate of the class based upon these predictions?

Improve this question
$\endgroup$

6 Answers 6

Reset to default
14
$\begingroup$

A good place to start is to get an overview of ensemble learning. Especially you'll want to look at boosting and bagging. Another method was that used by "The Ensemble" team in the Netflix Prize, is called either "blending" or feature stacking.

Then, just find some libraries that implement those and work from there. A quick googling turned up scikit and orange, both of which should have bagging and boosting (and they're both Python).

If beyond just using ensemble methods, you'd like to learn a bit of the theory, then I think this paper would be a good jumping off point (follow the references for the parts you're interested in).

Cheers.

Improve this answer
$\endgroup$
0
4
$\begingroup$

'Ensemble Methods in Data Mining: Improving Accuracy Through Combining Predictions', Seni and Elder -- Excellent reference on practical ensemble theory and implementation, but accompanying code is R based.

'Machine Learning: An Algorithmic Perspective,' S. Marsland -- Excellent Python based practical text, but not as dedicated to pure ensemble concepts as the first reference.

Improve this answer
$\endgroup$
3
$\begingroup$

Stumpy Joe Pete's response was perfect, but since you mentioned about a Python implementation, I wanted to mention the brew project from Universidade Federal de Pernambuco.

https://github.com/viisar/brew

from brew.base import Ensemble
from brew.base import EnsembleClassifier
from brew.combination import import Combiner

# create your Ensemble
clfs = your_list_of_classifiers # [clf1, clf2]
ens = Ensemble(classifiers = clfs)

# create your Combiner
# the rules can be 'majority_vote', 'max', 'min', 'mean' or 'median'
comb = Combiner(rule='majority_vote')

# now create your ensemble classifier
ensemble_clf = EnsembleClassifier(ensemble=ens, combiner=comb)
ensemble_clf.predict(X)

At this point, they have ensemble generation, combination, pruning and dynamic selection.

Limitations: classification only; no stacking in current public version; not much documentation.

Improve this answer
$\endgroup$
1
$\begingroup$

Scikit-learn ensembling guide provides bagging and boosting meta-classifiers and regressors. In addition, mlxtend library provides implementations of stacking meta-classifiers and regressors.

Improve this answer
$\endgroup$
0
$\begingroup$

Salford Systems has a software package called Random Forests that implements this for classification and regression tree ensembles. I don't have any free R packages to offer. I imagine they have a users manual that will explain their implementation. By analogy you could probably figure out how to do it for other ensemble methods.

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ There are many great R packages for ensembles of trees: e.g. randomForest (classic algorithm), party::cforest (random forest using conditional inference trees), gbm (gradient boosting of trees) to name a few. I read the OP as wanting to implement classifier/regression agnostic ensembles. The simplest procedure is of course to average predictions. $\endgroup$
    – B_Miner
    Commented Jul 20, 2012 at 17:03
  • $\begingroup$ @B_Miner It is nice to know that there are implementations available in R. Maybe somebody could explain to me why a specific implementation in Python is desirable (please excuse my ignorance about R). I read the the OP to want to know sources that describe how to implement the ensemble methods. The Salford package was one that I was aware of that might have some documentation. $\endgroup$
    – Michael R. Chernick
    Commented Jul 20, 2012 at 17:20
  • $\begingroup$ While based on the Freund and Schapire paper boosting works in general as far as I know the best results have come using tree classifiers. $\endgroup$
    – Michael R. Chernick
    Commented Jul 20, 2012 at 21:24
  • $\begingroup$ I personally get really good results by simple averaging of probabilities - but my domain is more interested in probabilities than picking a class label. $\endgroup$
    – B_Miner
    Commented Jul 21, 2012 at 0:33
  • $\begingroup$ @MichaelChernick If you're doing really intense predictive work (like... a Kaggle competition), you're not going to pick either boosting or random forests. You're going to want to aggregate as many possible models as will help you (which is generally a lot more than one). So, in that context, other ensemble methods are going to be important, even if random forests are way awesomer than anything else. $\endgroup$
    – Stumpy Joe Pete
    Commented Sep 19, 2012 at 2:44
0
$\begingroup$

I found this tutorial which was extremely helpful. It doesn't answer all the pieces but I think it is a great start to the discussion: http://vikparuchuri.com/blog/intro-to-ensemble-learning-in-r/

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.