Equal sized sets in cross-validation

Question

What is the advantage you get out of choosing equal sized training and test sets in cross-validation? Why don't you split your set in two sets of different size in 2-fold cross-validation for example?

Answer 1

• I'd describe the relevant splitting idea in CV rather as "for all surrogate models training sets have roughly equal size, and test sets have roughly equal size as well".

• 2-fold CV is the only CV scheme, where you have in addition to the "same size" property, both training sets and test sets independent of each other for the 2 (surrogate) models.

• With a 2-fold CV splitting, say, 30 : 70, you'd get a well trained trained (70 train) surrogate model that is not tested well (only 30 test cases) and a model trained with very few cases (less than half of the training sample size of the 1st surrogate model) that is however, tested with more cases. If you go for algorithm comparison this complicates things as the learning curve may be different for your algorithms.

---

You can of course do repeated runs of a set validation that splits 70:30. Just it's then not called cross validation, and of course, the training sets are not independent (just as the 5 x 2 CV training sets are not independent between the runs, just within each run)

---

Side note: Note that cross validation is used a lot for estimating performance of the model you obtain from a given training set, as opposed to comparison of training algorithms. In the first case, you want to have the training sets of the surrogate models as similar as possible (= highly dependent), in the second case you want to have them independent.

Answer 2

Assuming that you mean using CV in the training partition, with a left-out test partition that you only use for obtaining a final performance estimate: using a large test set reduces the chance obtaining good/bad test results just by chance, so increase the probability that your estimates are realistic (cf. e.g. Kaggle challenges, where a huge portion of the data is kept back for final tests).

If you instead just do CV with 2 equally sized partitions you probably use those results for parameter tuning/model selection, which can lead to overfitting when not having an additional left-out test set.

PS: why would you use differently sized CV partitions instead of equally sized ones in training?