1
$\begingroup$

I would like to do 10 times repeated 5-fold CV and compare results from more models. Here is one thing I don't understand:

Example of 5-fold CV:

myFolds <- createFolds(loan_data$loan_status, k = 5)

summary(myFolds)

Lenghts:

Fold1 - 2011,

Fold2 - 2012,....., Fold5 - 2012

Folds have ~ 2012 indexes which is ~ 20%

This should make 5 folds and I can use them in index argument of trainControl function:

myControl <- trainControl( method = "cv",
number = 5, summaryFunction = twoClassSummary, classProbs = TRUE, index = myFolds )

From documentation: index a list with elements for each resampling iteration. Each list element is a vector of integers corresponding to the rows used for training at that iteration.

So does it mean, that train function will in all 5 resamples train data on ~2011 rows and test on ~8044 rows? Should I use indexOut instead of index if I want 80% train and 20% test?

Second function :

myFolds <- createMultiFolds(loan_data$loan_status, 5, 10)

summary(myFolds)

Fold1.Rep1: 8047,.......

Fold5.Rep10: 8048

Folds have ~ 8048 indexes which is ~ 80% It seems like createMultiFolds creates folds for TRAINING sets while createFolds creates folds for TESTING sets. Am I right? In this example, question is same... Should I use index or indexOut?

myControl <- trainControl(method = "repeatedcv",
number = 5,repeats = 10, summaryFunction = twoClassSummary, classProbs = TRUE, index = myFolds #indexOut = myFolds )

Thank you for any explanation.

$\endgroup$

2 Answers 2

3
$\begingroup$

Here is my experience/summary:

if you want to return train set, you can createFolds(loan_data$loan_status, k = 5, returnTrain=TRUE)

internally, createMultiFolds(loan_data$loan_status, 5, 10) calls createFolds(returnTrain=TRUE)

when you use index in trainControl() the parameters number, repeats are ignored (https://github.com/topepo/caret/issues/584)

$\endgroup$
0
$\begingroup$

Best case you predefine your indices via createMultiFolds.

Its important that by default - if "cv" selected instead of "repeatedcv" or repeates not defined in trainControl - there won't be repeates.

So, you have to predefine you multifold indices then set the number of repeates in the trainControl.

k - number of the folds will be automatically recognized and used

t - times (repeates) must be defined

multiFoldIndices <- createMultiFolds(y_train,5,3)
caretListRes <- caretList(x_train,
                          y_train,
                          trControl = trainControl(method = "repeatedcv",
                                                   index = multiFoldIndices,
                                                   repeats = 3,
                                                   savePredictions = "final",
                                                   allowParallel = TRUE
                                                   ),
                          methodList = c("lm","rf"),
                          preProcess = c("center","scale")
                          )
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.