Cross Validation in small datasets

I have a really small dataset (124 samples) and I'd like to try out if I get some interesting results with some machine learning algorithms in R.

What I've done: I splitted my data set into 75% training and 25% test, and trained six diferent models with the structure similar as follows:

fitControl <- trainControl(## 10-fold CV
method = "repeatedcv",
number = 10,
## repeated five times
repeats = 5,
savePredictions = TRUE,
classProbs = TRUE,
summaryFunction = twoClassSummary)

model_SVM_P <- train(Group ~ ., data = training_set,
method = "svmPoly",
trControl = fitControl,
metric = "Accuracy",
verbose = FALSE,
tuneLength = 10)


However, I just started studying about machine learning and deep learning and the cross validation part is aways hard to understand.

The question is: In the code there is only the inner cross validation step, is it necessary to do an outer loop for cv? If yes, how can I code it?

EDIT:

Just an update that can be helpful if someone is passing by with the same problem:

I did this and worked fine:

folds <- createFolds(training_set$$Group,4) split_up <- lapply(folds, function(ind, dat) dat[ind,], dat = training_set) parms_list_SVM_P <- list() model_list_SVM_P <- list() for (i in 1:4) {model_SVM_P <- train(Group ~ ., data = split_up[[i]], method = "svmPoly", trControl = fitControl, metric = "Accuracy", verbose = FALSE, tuneLength = 10) model_list_SVM_P[[i]] <- model_SVM_P parms_list_SVM_P[[i]] <- model_SVM_P$$bestTune}


Now I'm proceding to further analysis.

If someone with more expertise find a mistake, please let me know.

This link helped me a lot: https://stackoverflow.com/questions/62183291/statistical-test-with-test-data/62193116#62193116

• You could also be interested in repeated random subsampling validation which can work well for small datasets Aug 20 '20 at 15:55
• Not sure if I understood.For example, in a 4-fold CV that would be create those folds randomly and more than one time? Like in the trainControl but for the outer CV? Aug 20 '20 at 16:51
• see this question for more info stats.stackexchange.com/questions/51416/… not sure if RRSV (also known as monte carlo cross validation) is implemnented in Caret, you might have to do some coding by yourself Aug 20 '20 at 18:54
• – Tim
Aug 21 '20 at 6:59

There are 2 points to consider:

• you need an outer validation step whenever you use the results of the inner validation to tune your model.
The reason is that tuning by a noisy performance estimate causes that estimate to become optimistically biased as a performance estimate.

• Your performance estimates are noisy because of the small number of test cases (look up binomial confidence intervals and measuring proportions). As an example, if you observe 3 misclassifications out of 31 tested cases this i s 90 % accuracy with a 95 % confidence interval 77 - 97 %.

In other words, unless the observed performance for your tuning grid has stark differences between the best and worst models, you cannot reliably choose based on so few tested cases.

What to do?

• (repeated) cross validation in the inner (optimization) validation gives you somewhat more reliable performance estimates. (Don't expect miracles, though.)

• Accuracy (or other proportions) are subject to very large variance uncertainty - they need very large test sample sizes to become certain. They have further characteristics that makes them not very well suited as target functional for optimization. See whether you can use a (strictly) proper scoring rule instead.

• so, for model selection the outer cv is needed. After I selected the best model, in the train/test with full dataset should be done again with, let's say a 4-fold nested repeated cv again or can I eliminate one cv (outer or inner) and do only a 4-fold nested cross validation for example? Aug 21 '20 at 17:12
• Nested CV describes already the combination of inner CV for hyperparameter tuning plus outer CV to estimate generalization error of the optimized model. Therefore, you don't need another "inner-inner" CV with nested CV. Aug 23 '20 at 18:09

In small datasets, such as your case, it is strongly advised because the train/test split can be noisy. Your performance estimates will be much more robust if you do outer CV. As far as I know, you need to code outer CV by yourself and use your code inside the loop as is. This may help.