I am performing 5-fold cross-validation on a relatively large data set and I have noticed that the validation error for each of the 5 training sets are very similar. So I guess, in this case, cross-validation is not very useful (it would be about the same as just using one training and test set). So I was wondering if I am working with a special case, or is this the case for all large data sets. I'm thinking that perhaps if you have enough training examples, the average cross-validation score would not be very different than the score for one training and test set. Is this intuitition correct?


It is certainly adds value to a single test because you get a stronger justification that your estimated accuracy is correct.

Large dataset certainly helps in making robust, accurate models though it won't bias the cross-validation on its own. The only possible problem you should check for is whether the set contains significant fraction of duplicated objects -- this may happen if the number of attributes is very small in comparison.

| cite | improve this answer | |
  • $\begingroup$ Thanks, it is actually a text dataset, where each example contains some text and tags associated with that text. In the training set association rules between text words and tag words are found and these are then used for prediction in the test set. There are very few duplicate examples $\endgroup$ – user1893354 Nov 3 '13 at 20:44
  • $\begingroup$ Having a larger number of folds reduces the variance of the estimates but increases the bias as the effective estimation sample size shrinks for a larger number of folds. It is always best to try several values for the number of folds to understand how your procedure behaves with certain data, I would think. $\endgroup$ – Jonas Striaukas Mar 19 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.