Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
A form of cross validation takes multiple subsets of a data and trains the model on them, then tests on the remaining subsets. Eventually, all subsets will be trained on.
Although I know that training once on the whole dataset will more likely result in overfitting than using cross validation, the end result is both methods result in the whole dataset being trained on. Why is cross validation better?
Cross validation is a way to deal with overfitting. In a simpler setup, you have a training and hold out sample. You train on a training, then test the model on the holdout. The problem was that people started using the results on (out-of-sample) holdout samples to select the models. This made holdout samples similar to training sample in many ways. You look at in-sample and out-of-sample fit metrics, then pick the best model. This is almost the same as fitting to the whole sample once (in-sample).
So, the trend now is to split into a training, cross-validation and test samples. You have this one test sample which is not used for model selection at all. It's your last bullet: once you selected the best model in-sample and out-of-sample, you test this one last selection on the test sample (which was not used so far) to validate that the model works.
I guess you are asking for something slightly different here: after you used cross validation (CV) for model training, evaluation, and selection, you train the best performing model type and hyperparameter set once again, using all training data. This model is the one you will use on your held back test set on once. So if I understand correctly what you want to point out is that only using CV and getting some results does not change anything in the training process of the chosen model, hence conceptually does not prevent any overfitting - and this is correct so far. But:
Your CV results give you more and better information about how different model types and hyperparameterizations perform on data the model has not seen before. Imagine only doing one training and one cross validation partition: you could be lucky (having the partitions separated in a way that from each you can predict the other well). If you use e.g. a 10 fold CV with 20 repeats you train your model 200 times and each time evaluate it on a portion of data it has not seen before - so this will be much more robust, as being lucky 200 times is less likely. If you have 200 such evaluation results for multiple types of models and multiple hyperparameter sets you have better information on which to choose the model you actually want to train once using all training data. This process ensures you choose something that will less likely overfit, therefore is a big advantage over using a single training+test partition - and I guess this is the difference you asked for in your question.
