0
$\begingroup$

I just learned of nested cross-validation and wanted to understand how my current approach is worse/ok.

Currently I would:

  1. Divide the data into a train/test set (80/20ish).

  2. Use k-fold cross-validation on the training set to tune my model.

  3. Refit the model on the training set and stop touching it. Get the score of the test set as my sense of how well the model will perform. Done.

What is the comparative advantage of using a nested cross-validation approach instead? Is it simply that I get to use my full dataset with nested cv (and maybe I get a distribution estimate instead of a point one)?

$\endgroup$
7
  • $\begingroup$ It's a more efficient use of your data. $\endgroup$ Nov 3 '20 at 13:49
  • $\begingroup$ Yes, the main point is to get a fair estimate of model performance by making use of the full dataset. Simply holding out one test set might make the estimate too optimistic $\endgroup$
    – doubllle
    Nov 3 '20 at 13:49
  • $\begingroup$ @doubllle How would the holdout result in overoptimism compared to nested cv? The holdout can be a significant chunk of data and has yet to ever be touched in this case. $\endgroup$
    – Josh
    Nov 3 '20 at 13:53
  • 1
    $\begingroup$ It would all depend on how confident you are that the hold out is truly a representative sample of the true distribution of your data. If your N is small, this is likely not the case. It could also underperform as well. Nested cross-validation will give you a confidence interval around your model performance and the model-building process as a whole. $\endgroup$ Nov 3 '20 at 14:00
  • 1
    $\begingroup$ To some extent this depends on how much data you have. "80/20ish" means how many exactly? If your 20% are a lot like e.g. n>=1000, I'd be fine with that. Also it depends on how computationally intensive your model fit is and whether nested CV is not too computing-time heavy. $\endgroup$ Nov 3 '20 at 14:02
0
$\begingroup$

I think the main advantage is that you will have more confidence in the score. Like with any other statistic, if you measure it many times, and average it, you reduce the variance of the estimation. But it depends a lot on the size of the data, of corurse.

You can check out the Python package that I built (mainly for my own projects) that performs nested cross validation:

https://github.com/JaimeArboleda/nestedcvtraining

It's the first package that I made, so any comment will be very appreciated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.