1
$\begingroup$

I have seen many examples on Nested CV where someone takes the entire dataset and performs nested CV on it.

My question is: for model comparison shouldn't we first split the original dataset into train and test once, and then perform Nested CV (so with both inner and outer loops) on the train set?

This way, once we have our best models resulting from the different Nested CV rounds for each candidate ML algorithm, we can compare them on the same original test set.

Improve this question
$\endgroup$
5
  • $\begingroup$ Usually people do CV when they do not have enough data for splitting it, so that may be the reason. $\endgroup$
    – user2974951
    Aug 28, 2019 at 5:53
  • $\begingroup$ Well but what about when doing model comparison? $\endgroup$
    – Odisseo
    Aug 28, 2019 at 5:54
  • $\begingroup$ Well, again, it would be nice, but if you do not have enough data, you will use the CV results to compare them. $\endgroup$
    – user2974951
    Aug 28, 2019 at 6:06
  • $\begingroup$ I never said I don’t have enough data. I’m just asking in an ideal setting. I have plenty of data. $\endgroup$
    – Odisseo
    Aug 28, 2019 at 7:49
  • $\begingroup$ In that case you should split the data and retain a test set. $\endgroup$
    – user2974951
    Aug 28, 2019 at 7:53

1 Answer 1

Reset to default
2
$\begingroup$

If you are applying model selection you will need an additional test set to perform out of sample performance estimation.

This is necessary due to degrading performance during your second step of inference (your model selection is inference).

You are allowed to use the test set only once to estimate the performance of your final estimator (e.g. ML algorithm).

Exception: If you are just interested in the best performing model you could theoretically leave this step out (no test set). However, in this case you never estimated the performance of your final classifier.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thank you. But what if, after I perform model selection, I want to give a final estimate for the final ML algorithm I selected as the best one? Should I have kept this original test set and not used it all along until this final test, or will it be ok to redo a new train-test split and use this final ML Algo against this new test set? $\endgroup$
    – Odisseo
    Aug 29, 2019 at 13:22
  • $\begingroup$ Yes, you are not allowed to use it until your final estimate and only once. Only this will be an unbiased estimate. $\endgroup$
    – user2672299
    Aug 29, 2019 at 13:45
  • $\begingroup$ You are not allowed to train, validate and select models and later draw a test set from this training data and retrain. $\endgroup$
    – user2672299
    Aug 29, 2019 at 13:46
  • $\begingroup$ Every decision - even if it is subconscious - made based on information from the test set influences the performance of your estimator on the test set. That is why you are not allowed to use it in anyway. $\endgroup$
    – user2672299
    Aug 29, 2019 at 13:49
  • $\begingroup$ datascience.stackexchange.com/a/18347/80280 $\endgroup$
    – user2672299
    Aug 30, 2019 at 6:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.