8
$\begingroup$

Could somebody provide a nice example code how to best implement an outer crossvalidation cycle using the caret package in R? The package provides a convenient trainControl() argument to ajust the inner crossvalidation. However I would like to embed this into multiple outer crossvalidation cycles to get a more stable estimate of the prediction performance of the estimated models!

$\endgroup$

1 Answer 1

9
$\begingroup$

Inner and outer CV are used to perform classifier selection not to get a better prediction on the estimate. To get a better estimate, do a repeated cv. So to perform a 10-repeates 5-fold CV use

trainControl(method = "repeatedcv",number = 5,
             ## repeated ten times
             repeats = 10)

But if what you really want is a nested CV, for example to select between a random forest or a svm) then as far as know you have to do the outer CV explicitly. What I did for an outer 5-fold, inner 10-fold was:

ntrain=length(ytrain)    
train.ext=createFolds(ytrain,k=5,returnTrain=TRUE)
test.ext=lapply(train.ext,function(x) (1:ntrain)[-x])

for (i in 1:5){
    model<-train(Class ~ ., data = training[train.ext[[i]]],
                 trControl=trainControl(method = "cv",number = 10),
                 ...
    ...
    }
$\endgroup$
3
  • 4
    $\begingroup$ I've never really seen a case where nested CV is worth the effort but I did put in the caret development list a few weeks ago. It may take a while. $\endgroup$
    – topepo
    Commented Dec 7, 2014 at 22:23
  • $\begingroup$ This is caret author's blog on nested cv: appliedpredictivemodeling.com/blog/2017/9/2/… $\endgroup$
    – zhanxw
    Commented May 18, 2018 at 16:12
  • $\begingroup$ @Jacques Wainer your statement that repeated cv is preferred to nested cv for getting a better estimate is very interesting - do you have any citation for that? does it refer to robustness, and how is this better addressed in repeated cv than nested cv? $\endgroup$
    – Agile Bean
    Commented Jul 16, 2021 at 9:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.