Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I often find myself training several different predictive models using caret in R. I'll train them all on the same cross validation folds, using caret::: createFolds, then choose the best model based on cross-validated error.

However, the median prediction from several models often outperforms the best single model on an independent test set. I'm thinking of writing some functions for stacking/ensembling caret models that were trained with the same cross-validation folds, for example by taking median predictions from each model on each fold, or by training a "meta-model."

Of course, this might require an outer cross-validation loop. Does anyone know of any existing packages/open source code for ensembling caret models (and possibly cross-validating those ensembles)?

share|improve this question
add comment

3 Answers

What you are looking for is called "model ensembling". A simple introductory tutorial with R code can be found here: http://viksalgorithms.blogspot.jp/2012/01/intro-to-ensemble-learning-in-r.html

share|improve this answer
    
Not to be nit picky, but "ensembling" is right in the title of my post. I'm very specifically looking for an R package for ensembling arbitrary models, which doesn't seem to exist. Thanks for posting the code, though. Maybe I'll write my own package! –  Zach Oct 15 '12 at 19:09
add comment

It looks like Max Kuhn actually started working on a package for ensembleling caret models, but hasn't had time to finish it yet. This is exactly what I was looking for. I hope the project gets finished one day!

share|improve this answer
add comment

I'm not quite sure what you are looking for but this might help: http://www.jstatsoft.org/v28/i05/paper

It is how to use multiple models in caret. The part you might be interested is section 5 on pg. 13.

share|improve this answer
    
What I'm looking for is a package that would take as an input a list of caret objects, and would then output the median, mean, or weighted mean average of their predictions. More advanced functionality might include optimizing the weights through nested-cross validation. –  Zach Oct 15 '12 at 19:11
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.