Cross Validation in small datasets

Question

I have a really small dataset (124 samples) and I'd like to try out if I get some interesting results with some machine learning algorithms in R.

What I've done: I splitted my data set into 75% training and 25% test, and trained six diferent models with the structure similar as follows:

fitControl <- trainControl(## 10-fold CV
                           method = "repeatedcv",
                           number = 10,
                           ## repeated five times
                           repeats = 5,
                           savePredictions = TRUE,
                           classProbs = TRUE,
                           summaryFunction = twoClassSummary)
    
model_SVM_P <- train(Group ~ ., data = training_set, 
                 method = "svmPoly", 
                 trControl = fitControl,
                 metric = "Accuracy",
                 verbose = FALSE,
                 tuneLength = 10)

However, I just started studying about machine learning and deep learning and the cross validation part is aways hard to understand.

The question is: In the code there is only the inner cross validation step, is it necessary to do an outer loop for cv? If yes, how can I code it?

EDIT:

Just an update that can be helpful if someone is passing by with the same problem:

I did this and worked fine:

folds <- createFolds(training_set$Group,4)
split_up <- lapply(folds, function(ind, dat) dat[ind,], dat = training_set)
parms_list_SVM_P <- list()
model_list_SVM_P <- list()
for (i in 1:4) {model_SVM_P <- train(Group ~ ., data = split_up[[i]], 
                 method = "svmPoly", 
                 trControl = fitControl,
                 metric = "Accuracy",
                 verbose = FALSE,
                 tuneLength = 10)
	model_list_SVM_P[[i]] <- model_SVM_P
	parms_list_SVM_P[[i]] <- model_SVM_P$bestTune}

Now I'm proceding to further analysis.

If someone with more expertise find a mistake, please let me know.

This link helped me a lot: https://stackoverflow.com/questions/62183291/statistical-test-with-test-data/62193116#62193116

Answer 1

There are 2 points to consider:

• you need an outer validation step whenever you use the results of the inner validation to tune your model.
  The reason is that tuning by a noisy performance estimate causes that estimate to become optimistically biased as a performance estimate.

• Your performance estimates are noisy because of the small number of test cases (look up binomial confidence intervals and measuring proportions). As an example, if you observe 3 misclassifications out of 31 tested cases this i s 90 % accuracy with a 95 % confidence interval 77 - 97 %.

  In other words, unless the observed performance for your tuning grid has stark differences between the best and worst models, you cannot reliably choose based on so few tested cases.

What to do?

• (repeated) cross validation in the inner (optimization) validation gives you somewhat more reliable performance estimates. (Don't expect miracles, though.)

• Accuracy (or other proportions) are subject to very large variance uncertainty - they need very large test sample sizes to become certain. They have further characteristics that makes them not very well suited as target functional for optimization. See whether you can use a (strictly) proper scoring rule instead.

Answer 2

In small datasets, such as your case, it is strongly advised because the train/test split can be noisy. Your performance estimates will be much more robust if you do outer CV. As far as I know, you need to code outer CV by yourself and use your code inside the loop as is. This may help.