I am performing 5-fold cross-validation on a relatively large data set and I have noticed that the validation error for each of the 5 training sets are very similar. So I guess, in this case, cross-validation is not very useful (it would be about the same as just using one training and test set). So I was wondering if I am working with a special case, or is this the case for all large data sets. I'm thinking that perhaps if you have enough training examples, the average cross-validation score would not be very different than the score for one training and test set. Is this intuitition correct?
$\begingroup$
$\endgroup$
asked Nov 3, 2013 at 16:36
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
It is certainly adds value to a single test because you get a stronger justification that your estimated accuracy is correct.
Large dataset certainly helps in making robust, accurate models though it won't bias the cross-validation on its own. The only possible problem you should check for is whether the set contains significant fraction of duplicated objects -- this may happen if the number of attributes is very small in comparison.
-
$\begingroup$ Thanks, it is actually a text dataset, where each example contains some text and tags associated with that text. In the training set association rules between text words and tag words are found and these are then used for prediction in the test set. There are very few duplicate examples $\endgroup$ Commented Nov 3, 2013 at 20:44
-
$\begingroup$ Having a larger number of folds reduces the variance of the estimates but increases the bias as the effective estimation sample size shrinks for a larger number of folds. It is always best to try several values for the number of folds to understand how your procedure behaves with certain data, I would think. $\endgroup$ Commented Mar 19, 2020 at 13:21