What is the advantage you get out of choosing equal sized training and test sets in cross-validation? Why don't you split your set in two sets of different size in 2-fold cross-validation for example?


2 Answers 2

  • I'd describe the relevant splitting idea in CV rather as "for all surrogate models training sets have roughly equal size, and test sets have roughly equal size as well".

  • 2-fold CV is the only CV scheme, where you have in addition to the "same size" property, both training sets and test sets independent of each other for the 2 (surrogate) models.

  • With a 2-fold CV splitting, say, 30 : 70, you'd get a well trained trained (70 train) surrogate model that is not tested well (only 30 test cases) and a model trained with very few cases (less than half of the training sample size of the 1st surrogate model) that is however, tested with more cases. If you go for algorithm comparison this complicates things as the learning curve may be different for your algorithms.

You can of course do repeated runs of a set validation that splits 70:30. Just it's then not called cross validation, and of course, the training sets are not independent (just as the 5 x 2 CV training sets are not independent between the runs, just within each run)

Side note: Note that cross validation is used a lot for estimating performance of the model you obtain from a given training set, as opposed to comparison of training algorithms. In the first case, you want to have the training sets of the surrogate models as similar as possible (= highly dependent), in the second case you want to have them independent.


Assuming that you mean using CV in the training partition, with a left-out test partition that you only use for obtaining a final performance estimate: using a large test set reduces the chance obtaining good/bad test results just by chance, so increase the probability that your estimates are realistic (cf. e.g. Kaggle challenges, where a huge portion of the data is kept back for final tests).

If you instead just do CV with 2 equally sized partitions you probably use those results for parameter tuning/model selection, which can lead to overfitting when not having an additional left-out test set.

PS: why would you use differently sized CV partitions instead of equally sized ones in training?

  • $\begingroup$ I'm preparing a talk regarding the 5x2cv paired t test as proposed by Dietterich (1997). I wasn't sure how to answer that question if someone asked. $\endgroup$
    – Trocadero
    May 11, 2016 at 15:46
  • $\begingroup$ OK, this seems to touch some more theoretical aspects of cross validation and its side effects. Maybe these newer and older other questions can help. IMHO some of their answers and links in the comments give good insights to the underlying concepts. $\endgroup$ May 11, 2016 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.