I would like to do 10 times repeated 5-fold CV and compare results from more models. Here is one thing I don't understand:
Example of 5-fold CV:
myFolds <- createFolds(loan_data$loan_status, k = 5)
Fold1 - 2011,
Fold2 - 2012,....., Fold5 - 2012
Folds have ~ 2012 indexes which is ~ 20%
This should make 5 folds and I can use them in index argument of trainControl function:
myControl <- trainControl( method = "cv",
number = 5, summaryFunction = twoClassSummary, classProbs = TRUE, index = myFolds )
From documentation: index a list with elements for each resampling iteration. Each list element is a vector of integers corresponding to the rows used for training at that iteration.
So does it mean, that train function will in all 5 resamples train data on ~2011 rows and test on ~8044 rows? Should I use indexOut instead of index if I want 80% train and 20% test?
Second function :
myFolds <- createMultiFolds(loan_data$loan_status, 5, 10)
Folds have ~ 8048 indexes which is ~ 80% It seems like createMultiFolds creates folds for TRAINING sets while createFolds creates folds for TESTING sets. Am I right? In this example, question is same... Should I use index or indexOut?
myControl <- trainControl(method = "repeatedcv",
number = 5,repeats = 10, summaryFunction = twoClassSummary, classProbs = TRUE, index = myFolds #indexOut = myFolds )
Thank you for any explanation.