# Cross-Validation gives different result on the same data

I have done Cross-Validation by crossval function in matlab on my data, but when I run the Cross-Validation many times, it give me a different results, so is that normal? Or, is something wrong going on?

I think that the initialization of certain parameters differ from time to time, also the data, which is divided by cross-validation is different. If so, which result of cross-validation must be taken, the best, worst, or the mean results?

• I'm not familiar with that particular function but most CV procedures randomly shuffle the data before splitting it up into folds. So you should expect to get slightly different answers each time. There may be a function argument which stops the random shuffling. Also you can repeat the CV multiple times and take the mean results, this is what the R package caret typically does.
– Jeff
Apr 1 '15 at 15:30
• @Jeff, there are a differnce in the results up to 15%, is this normal ? Apr 3 '15 at 11:21
• I think it would depend on the structure/size of the data. If the data was quite small then yes, I think there could be quite different results between runs.
– Jeff
Apr 3 '15 at 11:26
• Cross validation can gives different answer when run many times. See documentation here and corss validation Mar 27 '16 at 9:21
• To @jeff 's point, the issue with getting "different answers" with CV is finite data samples. It is only in theory that information approaches an asymptotic limit. However, with some software you are able to run and rerun the CV on the same folds which would control for the "different answers each time." Mar 27 '16 at 10:28