-1
$\begingroup$

Every time the cross-validation is run, the dataset is partitioned into k even groups randomly. That means every time the result of cross-validation can be different, so which one should we take to determine the optimal parameters? Is it necessary to go through all the possibilities of the partitioning?

Many thanks in advance!

$\endgroup$
-4
$\begingroup$

Decide one partitioning and stick to it. Calculating the error again and again is more likely to give you a better result on your validation data but not on the test data on which it would be tested later

$\endgroup$
  • $\begingroup$ That's fine.Welcome to Stats Exchange ! $\endgroup$ – sww May 3 '18 at 15:35
5
$\begingroup$

Is it necessary to go through all the possibilities of the partitioning?

No it's not necessary to go through every possibility of the partitioning. You may do repeated k-fold cross validation (it takes longer), but it's perfectly acceptable to do one iteration of k-fold CV.

which one should we take to determine the optimal parameters?

You can take the average of all k CV error rates to get your estimate of out-of-sample error.

See this interesting analysis to answer most of your questions in more detail and with empirical evidence.

$\endgroup$
  • $\begingroup$ You should not do the same process again and again to get a better result as repeating will make it likely you will get something better by chance statistically. $\endgroup$ – sww May 3 '18 at 14:52
  • 5
    $\begingroup$ @sww Repeating the k-fold process and taking the average of all the repeats is done to reduce the variance of the estimated error. In a given cv estimate, you may very well get an unrealistically good or bad error by chance. That's why you take the average of many results, to reduce the influence of any single result. $\endgroup$ – timwiz May 3 '18 at 15:09
  • $\begingroup$ Thats the reason k fold is done. I am talking about repeating k fold multiple times. $\endgroup$ – sww May 3 '18 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.