0
$\begingroup$

A form of cross validation takes multiple subsets of a data and trains the model on them, then tests on the remaining subsets. Eventually, all subsets will be trained on.

Although I know that training once on the whole dataset will more likely result in overfitting than using cross validation, the end result is both methods result in the whole dataset being trained on. Why is cross validation better?

$\endgroup$
  • $\begingroup$ This is not a duplicate. I even explained what cross validation is the details. I'm asking why it is better. $\endgroup$ – StackOverflowOfficial Jun 14 '16 at 20:39
  • 1
    $\begingroup$ This question doesn't make sense. You state that using the whole dataset "will more likely result in overfitting than using cross validation", but then ask why cross validation is better. Are you wondering if overfitting is a good thing? $\endgroup$ – gung - Reinstate Monica Jun 14 '16 at 21:16
  • $\begingroup$ @Goldname I'm not sure I picked the best redirect target, but the answers there do cover the issue of what the point of cross-validation is, which is what this question is, so far as I can tell. $\endgroup$ – Kodiologist Jun 14 '16 at 21:30
  • $\begingroup$ @gung To the extent that this question doesn't make sense, it does so due to a misunderstanding that might be the focus of an answer. The core question here seems to be "how come cross-validation is still effective, even though it ultimately uses the entire dataset?" which is arguably a distinct question in its own right. (Though I'd agree that several of the answers to the linked question really also answer this one, I do think the question itself is distinctive.) $\endgroup$ – Silverfish Jun 14 '16 at 23:06
  • $\begingroup$ @Silverfish, if it were edited to be more coherent (eg, like you suggest), I'd be willing to leave / re- open. At present, I think this should probably be closed. $\endgroup$ – gung - Reinstate Monica Jun 14 '16 at 23:19
4
$\begingroup$

Cross validation is a way to deal with overfitting. In a simpler setup, you have a training and hold out sample. You train on a training, then test the model on the holdout. The problem was that people started using the results on (out-of-sample) holdout samples to select the models. This made holdout samples similar to training sample in many ways. You look at in-sample and out-of-sample fit metrics, then pick the best model. This is almost the same as fitting to the whole sample once (in-sample).

So, the trend now is to split into a training, cross-validation and test samples. You have this one test sample which is not used for model selection at all. It's your last bullet: once you selected the best model in-sample and out-of-sample, you test this one last selection on the test sample (which was not used so far) to validate that the model works.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ But why is cross validation better? $\endgroup$ – StackOverflowOfficial Jun 14 '16 at 20:38
  • 3
    $\begingroup$ Do you understand the overfitting issue? That's where the answer is. $\endgroup$ – Aksakal Jun 14 '16 at 20:49
  • $\begingroup$ Once you're all done with the cross validation, you would have basically overfit to your dataset $\endgroup$ – StackOverflowOfficial Jun 15 '16 at 1:30
  • 2
    $\begingroup$ It's like why do we need seat belts if one can kill himself even with it? Well, for 10 idiots who overfit with cross validation, there'll be one who'll use it to avoid this trap. I came to cross validation on my own, we were not trained on this back in the days. Once I realized that my out-of-sample tests are not that different from in-sample if they're used for model selection. Then I saw what people were doing with cross validation, and adopted it in my practice. $\endgroup$ – Aksakal Jun 15 '16 at 2:14
  • 2
    $\begingroup$ You are asking to close this question, effectively $\endgroup$ – Aksakal Jun 15 '16 at 11:18
1
$\begingroup$

I guess you are asking for something slightly different here: after you used cross validation (CV) for model training, evaluation, and selection, you train the best performing model type and hyperparameter set once again, using all training data. This model is the one you will use on your held back test set on once. So if I understand correctly what you want to point out is that only using CV and getting some results does not change anything in the training process of the chosen model, hence conceptually does not prevent any overfitting - and this is correct so far. But:

Your CV results give you more and better information about how different model types and hyperparameterizations perform on data the model has not seen before. Imagine only doing one training and one cross validation partition: you could be lucky (having the partitions separated in a way that from each you can predict the other well). If you use e.g. a 10 fold CV with 20 repeats you train your model 200 times and each time evaluate it on a portion of data it has not seen before - so this will be much more robust, as being lucky 200 times is less likely. If you have 200 such evaluation results for multiple types of models and multiple hyperparameter sets you have better information on which to choose the model you actually want to train once using all training data. This process ensures you choose something that will less likely overfit, therefore is a big advantage over using a single training+test partition - and I guess this is the difference you asked for in your question.

| cite | improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.