21
$\begingroup$

I often find myself training several different predictive models using caret in R. I'll train them all on the same cross validation folds, using caret::: createFolds, then choose the best model based on cross-validated error.

However, the median prediction from several models often outperforms the best single model on an independent test set. I'm thinking of writing some functions for stacking/ensembling caret models that were trained with the same cross-validation folds, for example by taking median predictions from each model on each fold, or by training a "meta-model."

Of course, this might require an outer cross-validation loop. Does anyone know of any existing packages/open source code for ensembling caret models (and possibly cross-validating those ensembles)?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
20
$\begingroup$

It looks like Max Kuhn actually started working on a package for ensembleling caret models, but hasn't had time to finish it yet. This is exactly what I was looking for. I hope the project gets finished one day!

edit: I wrote my own package to do this: caretEnsemble

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Excellent work on this package! $\endgroup$
    – mjc
    Feb 24, 2016 at 16:23
8
$\begingroup$

What you are looking for is called "model ensembling". A simple introductory tutorial with R code can be found here: http://viksalgorithms.blogspot.jp/2012/01/intro-to-ensemble-learning-in-r.html

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ Not to be nit picky, but "ensembling" is right in the title of my post. I'm very specifically looking for an R package for ensembling arbitrary models, which doesn't seem to exist. Thanks for posting the code, though. Maybe I'll write my own package! $\endgroup$
    – Zach
    Oct 15, 2012 at 19:09
1
$\begingroup$

I'm not quite sure what you are looking for but this might help: http://www.jstatsoft.org/v28/i05/paper

It is how to use multiple models in caret. The part you might be interested is section 5 on pg. 13.

Improve this answer
$\endgroup$
1
  • $\begingroup$ What I'm looking for is a package that would take as an input a list of caret objects, and would then output the median, mean, or weighted mean average of their predictions. More advanced functionality might include optimizing the weights through nested-cross validation. $\endgroup$
    – Zach
    Oct 15, 2012 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.