0
$\begingroup$

I would like to do 10 times repeated 5-fold CV and compare results from more models. Here is one thing I don't understand:

Example of 5-fold CV:

myFolds <- createFolds(loan_data$loan_status, k = 5)

summary(myFolds)

Lenghts:

Fold1 - 2011,

Fold2 - 2012,....., Fold5 - 2012

Folds have ~ 2012 indexes which is ~ 20%

This should make 5 folds and I can use them in index argument of trainControl function:

myControl <- trainControl( method = "cv",
number = 5, summaryFunction = twoClassSummary, classProbs = TRUE, index = myFolds )

From documentation: index a list with elements for each resampling iteration. Each list element is a vector of integers corresponding to the rows used for training at that iteration.

So does it mean, that train function will in all 5 resamples train data on ~2011 rows and test on ~8044 rows? Should I use indexOut instead of index if I want 80% train and 20% test?

Second function :

myFolds <- createMultiFolds(loan_data$loan_status, 5, 10)

summary(myFolds)

Fold1.Rep1: 8047,.......

Fold5.Rep10: 8048

Folds have ~ 8048 indexes which is ~ 80% It seems like createMultiFolds creates folds for TRAINING sets while createFolds creates folds for TESTING sets. Am I right? In this example, question is same... Should I use index or indexOut?

myControl <- trainControl(method = "repeatedcv",
number = 5,repeats = 10, summaryFunction = twoClassSummary, classProbs = TRUE, index = myFolds #indexOut = myFolds )

Thank you for any explanation.

$\endgroup$
2
$\begingroup$

Here is my experience/summary:

if you want to return train set, you can createFolds(loan_data$loan_status, k = 5, returnTrain=TRUE)

internally, createMultiFolds(loan_data$loan_status, 5, 10) calls createFolds(returnTrain=TRUE)

when you use index in trainControl() the parameters number, repeats are ignored (https://github.com/topepo/caret/issues/584)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.