It seems that similar arguments can be made for using nested cross-validation instead of a simple hold-out test set, as the arguments for using cross-validation instead of a single validation set. The main argument is that using cross-validation leads to a better approximation of out-of-sample model performance. The same should hold for using nested cross-validation instead of a single test set.

It seems that nested cross-validation would be particularly useful if you suspect there are covariate shifts in the data.

Is the main reason why nested cross-validation is not widely used, then, mostly computational, as it would require both significantly more computation time and careful thinking with regards to implementation? Am I correct that nested cross-validation should be strongly considered if you suspect there may be covariate shifts in the data, and so the performance on the hold-out test set may be a poor estimate for generalized out-of-sample performance of the model?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.