4
$\begingroup$

Is it valid to do a k-fold CV in the outer loop of a nested CV but use the holdout method in the inner loop to avoid computational complexity?

My guess is that one could leave one of the k sets out in the inner CV loop so that he could run a k-1 cross validation on the sets left where each fold would have a different validation set.

For example, if one is running a 4-fold CV in the outer loop then the dataset would initially be split like this (each number denotes the respective fold):

1st set

  • training set: 1-2-3
  • test set: 4

2nd set

  • training set: 2-3-4
  • test set: 1

3rd set

  • training set: 1-3-4
  • test set: 2

4th set

  • training set: 1-2-4
  • test set: 3

Then, for the 3 fold CV the data would look like this:

1st set

  • training set: 3-4
  • validation set: 2

2nd set

  • training set: 2-4
  • validation set: 3

3rd set

  • training set: 1-3
  • validation set: 4

So, to tune the set of hyperparameters one could test the model using the holdout method in each of the 3 sets and then just use the hyperparameters that performed better (on average) to test the model in the outer CV loop.

I assume that this is of course less accurate than a normal nested k-fold cross validation especially given low values of k. However, for higher values of k and bigger datasets would this result in a good trade-off between accuracy and computational complexity?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.