1
$\begingroup$

I have done Cross-Validation by crossval function in matlab on my data, but when I run the Cross-Validation many times, it give me a different results, so is that normal? or something wrong is going on?

I think that the initialization of certain parameters differ fom time to time, also the data, which divided by cross-validation is diffe. if that so, which result of cross-validation must be taken? the best, worst resulrts or the mean results?

$\endgroup$
  • $\begingroup$ I'm not familiar with that particular function but most CV procedures randomly shuffle the data before splitting it up into folds. So you should expect to get slightly different answers each time. There may be a function argument which stops the random shuffling. Also you can repeat the CV multiple times and take the mean results, this is what the R package caret typically does. $\endgroup$ – Jeff Apr 1 '15 at 15:30
  • $\begingroup$ @Jeff, there are a differnce in the results up to 15%, is this normal ? $\endgroup$ – user2162652 Apr 3 '15 at 11:21
  • $\begingroup$ I think it would depend on the structure/size of the data. If the data was quite small then yes, I think there could be quite different results between runs. $\endgroup$ – Jeff Apr 3 '15 at 11:26
  • $\begingroup$ To @jeff 's point, the issue with getting "different answers" with CV is finite data samples. It is only in theory that information approaches an asymptotic limit. However, with some software you are able to run and rerun the CV on the same folds which would control for the "different answers each time." $\endgroup$ – Mike Hunter Mar 27 '16 at 10:28
1
$\begingroup$

Cross validation (usually) works by repeatedly selecting a subset of the data to leave out, fitting a model using the data that was not left out and then evaluating the estimated model using the left out data.

If the subset of the data that is left out is selected at random, then the results may differ from run to run. You can try setting a seed if you want to get the same results every time you run cross validation.

As for the question of which result should be taken, you probably want the mean.

$\endgroup$
0
$\begingroup$

Cross validation can gives different answer when run many times. See documentation here and corss validation

$\endgroup$
  • $\begingroup$ Rather than simply sharing a link, it would be much more helpful to expand on the key points made in them. $\endgroup$ – Mike Hunter Mar 27 '16 at 10:18
  • 1
    $\begingroup$ I agree with @DJohnson. Links can as well be removed in the future, so this will make the answer pretty useless. A good practice is to provide and explain the key points, linking then to external sources for further infos. $\endgroup$ – AlessioX Mar 27 '16 at 10:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.