Is it valid to do a k-fold CV in the outer loop of a nested CV but use the holdout method in the inner loop to avoid computational complexity?

My guess is that one could leave one of the k sets out in the inner CV loop so that he could run a k-1 cross validation on the sets left where each fold would have a different validation set.

For example, if one is running a 4-fold CV in the outer loop then the dataset would initially be split like this (each number denotes the respective fold):

1st set

  • training set: 1-2-3
  • test set: 4

2nd set

  • training set: 2-3-4
  • test set: 1

3rd set

  • training set: 1-3-4
  • test set: 2

4th set

  • training set: 1-2-4
  • test set: 3

Then, for the 3 fold CV the data would look like this:

1st set

  • training set: 3-4
  • validation set: 2

2nd set

  • training set: 2-4
  • validation set: 3

3rd set

  • training set: 1-3
  • validation set: 4

So, to tune the set of hyperparameters one could test the model using the holdout method in each of the 3 sets and then just use the hyperparameters that performed better (on average) to test the model in the outer CV loop.

I assume that this is of course less accurate than a normal nested k-fold cross validation especially given low values of k. However, for higher values of k and bigger datasets would this result in a good trade-off between accuracy and computational complexity?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.