# Cross-Validation gives different result on the same data

I have done Cross-Validation by crossval function in matlab on my data, but when I run the Cross-Validation many times, it give me a different results, so is that normal? or something wrong is going on?

I think that the initialization of certain parameters differ fom time to time, also the data, which divided by cross-validation is diffe. if that so, which result of cross-validation must be taken? the best, worst resulrts or the mean results?

• I'm not familiar with that particular function but most CV procedures randomly shuffle the data before splitting it up into folds. So you should expect to get slightly different answers each time. There may be a function argument which stops the random shuffling. Also you can repeat the CV multiple times and take the mean results, this is what the R package caret typically does. – Jeff Apr 1 '15 at 15:30
• @Jeff, there are a differnce in the results up to 15%, is this normal ? – user2162652 Apr 3 '15 at 11:21
• I think it would depend on the structure/size of the data. If the data was quite small then yes, I think there could be quite different results between runs. – Jeff Apr 3 '15 at 11:26
• To @jeff 's point, the issue with getting "different answers" with CV is finite data samples. It is only in theory that information approaches an asymptotic limit. However, with some software you are able to run and rerun the CV on the same folds which would control for the "different answers each time." – Mike Hunter Mar 27 '16 at 10:28