5
$\begingroup$

I have a really small dataset (124 samples) and I'd like to try out if I get some interesting results with some machine learning algorithms in R.

What I've done: I splitted my data set into 75% training and 25% test, and trained six diferent models with the structure similar as follows:

fitControl <- trainControl(## 10-fold CV
                           method = "repeatedcv",
                           number = 10,
                           ## repeated five times
                           repeats = 5,
                           savePredictions = TRUE,
                           classProbs = TRUE,
                           summaryFunction = twoClassSummary)
    
model_SVM_P <- train(Group ~ ., data = training_set, 
                 method = "svmPoly", 
                 trControl = fitControl,
                 metric = "Accuracy",
                 verbose = FALSE,
                 tuneLength = 10)

However, I just started studying about machine learning and deep learning and the cross validation part is aways hard to understand.

The question is: In the code there is only the inner cross validation step, is it necessary to do an outer loop for cv? If yes, how can I code it?

EDIT:

Just an update that can be helpful if someone is passing by with the same problem:

I did this and worked fine:

folds <- createFolds(training_set$Group,4)
split_up <- lapply(folds, function(ind, dat) dat[ind,], dat = training_set)
parms_list_SVM_P <- list()
model_list_SVM_P <- list()
for (i in 1:4) {model_SVM_P <- train(Group ~ ., data = split_up[[i]], 
                 method = "svmPoly", 
                 trControl = fitControl,
                 metric = "Accuracy",
                 verbose = FALSE,
                 tuneLength = 10)
	model_list_SVM_P[[i]] <- model_SVM_P
	parms_list_SVM_P[[i]] <- model_SVM_P$bestTune}

Now I'm proceding to further analysis.

If someone with more expertise find a mistake, please let me know.

This link helped me a lot: https://stackoverflow.com/questions/62183291/statistical-test-with-test-data/62193116#62193116

$\endgroup$
4
  • 1
    $\begingroup$ You could also be interested in repeated random subsampling validation which can work well for small datasets $\endgroup$
    – MikeKatz45
    Aug 20 '20 at 15:55
  • $\begingroup$ Not sure if I understood.For example, in a 4-fold CV that would be create those folds randomly and more than one time? Like in the trainControl but for the outer CV? $\endgroup$ Aug 20 '20 at 16:51
  • 2
    $\begingroup$ see this question for more info stats.stackexchange.com/questions/51416/… not sure if RRSV (also known as monte carlo cross validation) is implemnented in Caret, you might have to do some coding by yourself $\endgroup$
    – MikeKatz45
    Aug 20 '20 at 18:54
  • 1
    $\begingroup$ Check stats.stackexchange.com/questions/tagged/… $\endgroup$
    – Tim
    Aug 21 '20 at 6:59
2
$\begingroup$

There are 2 points to consider:

  • you need an outer validation step whenever you use the results of the inner validation to tune your model.
    The reason is that tuning by a noisy performance estimate causes that estimate to become optimistically biased as a performance estimate.

  • Your performance estimates are noisy because of the small number of test cases (look up binomial confidence intervals and measuring proportions). As an example, if you observe 3 misclassifications out of 31 tested cases this i s 90 % accuracy with a 95 % confidence interval 77 - 97 %.

    In other words, unless the observed performance for your tuning grid has stark differences between the best and worst models, you cannot reliably choose based on so few tested cases.

What to do?

  • (repeated) cross validation in the inner (optimization) validation gives you somewhat more reliable performance estimates. (Don't expect miracles, though.)

  • Accuracy (or other proportions) are subject to very large variance uncertainty - they need very large test sample sizes to become certain. They have further characteristics that makes them not very well suited as target functional for optimization. See whether you can use a (strictly) proper scoring rule instead.

$\endgroup$
2
  • $\begingroup$ so, for model selection the outer cv is needed. After I selected the best model, in the train/test with full dataset should be done again with, let's say a 4-fold nested repeated cv again or can I eliminate one cv (outer or inner) and do only a 4-fold nested cross validation for example? $\endgroup$ Aug 21 '20 at 17:12
  • 1
    $\begingroup$ Nested CV describes already the combination of inner CV for hyperparameter tuning plus outer CV to estimate generalization error of the optimized model. Therefore, you don't need another "inner-inner" CV with nested CV. $\endgroup$ Aug 23 '20 at 18:09
2
$\begingroup$

In small datasets, such as your case, it is strongly advised because the train/test split can be noisy. Your performance estimates will be much more robust if you do outer CV. As far as I know, you need to code outer CV by yourself and use your code inside the loop as is. This may help.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.