1
$\begingroup$

It seems that similar arguments can be made for using nested cross-validation instead of a simple hold-out test set, as the arguments for using cross-validation instead of a single validation set. The main argument is that using cross-validation leads to a better approximation of out-of-sample model performance. The same should hold for using nested cross-validation instead of a single test set.

It seems that nested cross-validation would be particularly useful if you suspect there are covariate shifts in the data.

Is the main reason why nested cross-validation is not widely used, then, mostly computational, as it would require both significantly more computation time and careful thinking with regards to implementation? Am I correct that nested cross-validation should be strongly considered if you suspect there may be covariate shifts in the data, and so the performance on the hold-out test set may be a poor estimate for generalized out-of-sample performance of the model?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.