2
$\begingroup$

Say you want to tune several parameters of your model using $N$ data. What you usually do is splitting your $N$ data into 3 sets:

  • learning set: used to build your model;
  • validation set: used to select the best parameters of your model;
  • test set: used to test your model with the parameters selected on the validation set.

If you are using a CV procedure, the learning set and validation set are in fact the same and thus you only have a learning set + a test set. But my problem is that a cross-validation procedure does not avoid the appearance of overfitting. Thus, you could keep the 3 sets even if one uses CV. In that way, you use CV on the learning set to learn the parameters, you determine when to stop the learning by looking at the error on the validation set and then you test your parameters on the test set. Is that correct?

$\endgroup$

1 Answer 1

1
$\begingroup$

I think you are confused about what cross validation is and why you use it. Quite simply, cross validation in this context is just repeated use of the learning and validation sets you describe above. Instead of splitting your data once into learning and validation you do it multiple times and take the average of your results to decrease the variance associated with randomly splitting your data.

So cross validation isn’t any different than the process you describe above and saying “does not avoid the appearance of over fitting” is incorrect

$\endgroup$
3
  • $\begingroup$ But to me the issue is that by using CV, at some point, the data which was in the validation set is going to be used to learn the parameters because you do CV. So why you cannot do something like: you do CV on the learning, you compute the error on the validation set to know when to stop reducing the error on the CV and then test the parameters obtained on the test set? $\endgroup$
    – Akusa
    Sep 1, 2021 at 16:11
  • $\begingroup$ You’re misunderstanding. In CV, you train several models. No model is trained and validated on the same data. $\endgroup$ Sep 1, 2021 at 17:00
  • $\begingroup$ @AryaMcCarthy When we use CV, we minimize the average of the CV error so, in fact, we are somehow training on the same data, right? $\endgroup$
    – Akusa
    Sep 1, 2021 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.