1
$\begingroup$

I know there are many questions about cross validation but I couldn't find that generally describes all aspects (I have in my mind). In How to evaluate the final model after k-fold cross-validation it is described that cross validation is used to select the best model. I wonder about the following:

  1. Why are different models not just trained on the same data set and tested on the same separated dataset? Why the effort for using a different data set for each model?
  2. The same applies when searching the best parameters for a single model (like in grid search).

Can you please clear my questions?

CV

$\endgroup$
  • 1
    $\begingroup$ In my opinion, train/test approach gives you a point estimation which may be terribly wrong due to chance, but cross-validation is giving a distribution. Which is always better. $\endgroup$ – German Demidov Oct 9 '19 at 8:08
  • $\begingroup$ Thank you. But how exactly is it used? $\endgroup$ – Ben Oct 9 '19 at 8:10
  • $\begingroup$ Well, you select the fold-size according to your sample size, run cross-validation, look at the distribution of test performance and make conclusions =) like, how stable is the performance, what is the expected performance, etc. I think there are plenty of manuals on this. $\endgroup$ – German Demidov Oct 9 '19 at 8:27
  • 1
    $\begingroup$ Why are different models not just trained on the same data set and tested on the same separated dataset? Why the effort for using a different data set for each model? - I think you might be wrong here. Practitioners do train different models on the same data set and compare their performances . That's the approach when you want to select the best model for the given data. $\endgroup$ – naive Oct 9 '19 at 10:48
  • 1
    $\begingroup$ @GermanDemidov CV also returns point estimates, which may be even "wronger", since they are estimated on less data. Would you rather have 5 estimates of the mean from 5 smaller samples or 1 estimate from a larger sample? OP: what naive said, different models are built on the same data and tested on the same data. $\endgroup$ – user2974951 Oct 9 '19 at 11:03
1
$\begingroup$

Writing an answer as answering in the comments is not really desirable.

Why are different models not just trained on the same data set and tested on the same separated dataset? Why the effort for using a different data set for each model?

  • I think you might be wrong here. Practitioners do train different models on the same data set and compare their performances . That's the approach when you want to select the best model for the given data.

How to perform cross-validation?

Have a look here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.