1
$\begingroup$

I have done Cross-Validation by crossval function in matlab on my data, but when I run the Cross-Validation many times, it give me a different results, so is that normal? Or, is something wrong going on?

I think that the initialization of certain parameters differ from time to time, also the data, which is divided by cross-validation is different. If so, which result of cross-validation must be taken, the best, worst, or the mean results?

$\endgroup$
5
  • $\begingroup$ I'm not familiar with that particular function but most CV procedures randomly shuffle the data before splitting it up into folds. So you should expect to get slightly different answers each time. There may be a function argument which stops the random shuffling. Also you can repeat the CV multiple times and take the mean results, this is what the R package caret typically does. $\endgroup$
    – Jeff
    Apr 1 '15 at 15:30
  • $\begingroup$ @Jeff, there are a differnce in the results up to 15%, is this normal ? $\endgroup$ Apr 3 '15 at 11:21
  • $\begingroup$ I think it would depend on the structure/size of the data. If the data was quite small then yes, I think there could be quite different results between runs. $\endgroup$
    – Jeff
    Apr 3 '15 at 11:26
  • $\begingroup$ Cross validation can gives different answer when run many times. See documentation here and corss validation $\endgroup$ Mar 27 '16 at 9:21
  • $\begingroup$ To @jeff 's point, the issue with getting "different answers" with CV is finite data samples. It is only in theory that information approaches an asymptotic limit. However, with some software you are able to run and rerun the CV on the same folds which would control for the "different answers each time." $\endgroup$ Mar 27 '16 at 10:28
1
$\begingroup$

Cross validation (usually) works by repeatedly selecting a subset of the data to leave out, fitting a model using the data that was not left out and then evaluating the estimated model using the left out data.

If the subset of the data that is left out is selected at random, then the results may differ from run to run. You can try setting a seed if you want to get the same results every time you run cross validation.

As for the question of which result should be taken, you probably want the mean.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.