There are different methods for performing the outer part of nested cross-validation (i.e. outer-cross-validation) for estimating overall generalised expected classification accuracy, e.g.:

  1. Use the nested outer n-fold method. In the n-fold method, the dataset is split into 'n' subsets, where n-1 subsets are used for training, and the remaining 1 subset is used for testing, which is repeated n times to cover all data.

  2. Use nested outer randomisation of the dataset. In this method, the dataset is randomised each time, and split according to a given training/testing ratio (e.g. 80/20), which is repeated 'n' times.

In both cases, the training sets are passed to an inner-cross-validation procedure (e.g. grid search model selection function) while the testing set is used to estimate the performance of the cross-validated model.

My question is, which method is it better to use, does it matter and why? To be clear, I am asking specifically about the outer-cross-validation, not the inner-cross-validation, which is used for model selection, rather than generalisation of performance estimates.

  • $\begingroup$ Do you have links to the references where you saw each method? $\endgroup$ – Dan Mar 27 '18 at 8:14
  • $\begingroup$ They are taken from peer reviewed papers on protein interactions however sorry I do not have the references immediately available $\endgroup$ – Aalawlx Mar 27 '18 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.