1
$\begingroup$

I am performing nested cross-validation, and I know that the idea behind it is to see how the model generalizes. For that, we don't only shuffle the training data but we also do shuffle the testing data.

Having this said, should nested cv be done on the whole data or just on the training part (like split the the data into 80% training and 20% testing and apply nested cv only on the 80% training). However I feel this is illogical since the whole idea is to see how well it generalizes.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The key idea is that cross-validation is not a method for finding out how well a model generalizes, but how well a procedure for fitting a model generalizes. So if your model has hyper-parameters that are tuned via cross-validation, then that is an integral part of the model fitting procedure and needs to be included in the outer cross-validation as well, as otherwise it doesn't account for the uncertainty in tuning the hyper-parameters.

See my answer here to a question about cross-validation based feature selection, for a simple example where if you don't use nested cross-validation you get a very optimistically biased performance estimate.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you. Yes I understand you very well. What I mean here by 'model' rather the algorithm used (Lasso Regression, Ridge, Decision Tree), and not the set of hyper parameters. For the hyper parameter tuning, I am tuning in the inner loop and retraining my data with the winning parameters from the inner loop in the outer loop. My question is really whether I must do nested cv on the whole data. $\endgroup$
    – Perl Del Rey
    Sep 20, 2019 at 7:43
  • $\begingroup$ choosing a model is equivalent to hyper-parameter tuning, in that you effectively have a hyper-parameter which tells you which model to use. Choosing which model to use is effectively part of the model generating procedure that you use to come up with the final model. Whether you need to do it depends on the size of the dataset and how many models you have to choose from (c.f. arxiv.org/abs/1809.09446) $\endgroup$
    – Dikran Marsupial
    Sep 20, 2019 at 7:47
  • $\begingroup$ The feature selection example I mentioned is quite apposite as there you are effectively choosing between models with different architectures (attribute sets), rather than explicit hyper-parameter selection. The key point is that if you make a data-based choice about your model, then you run the risk of biasing your performance estimation if you don't include that selection procedure within the cross-validation. $\endgroup$
    – Dikran Marsupial
    Sep 20, 2019 at 7:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.