0
$\begingroup$

I am the kind that understands much easier on numerical examples.

Assume we will use 5-fold cross-validation. Can you calculate number of examples in training, test and validation sets for a data that have 100 examples in total?

Thanks a lot.

$\endgroup$
  • $\begingroup$ Give us your go on the numbers, and we'll correct it ;-) $\endgroup$ – cbeleites supports Monica Feb 18 '16 at 11:38
  • $\begingroup$ Ok. Assume training, test, validation are the number of examples in training, test, and validation sets. My guess is test=validation=100/5=20. Remaining becomes training set: training=100-2*20=60. But I am not sure whether test and validation should be equal. $\endgroup$ – an apprentice Feb 18 '16 at 13:36
1
$\begingroup$

k-fold cross validaton splits into 2 sets, i.e. for 100 cases 5-fold CV

  • 5 x (20 validation/optimization + 80 trainig), or
  • 5 x (20 test/final validation + 80 optimized training)

Splitting into 3 (training + validation/optimization + test/final validation) sets e.g. by nesting k-fold cross validation. You can choose the inner and outer validation schemes independently, including k-fold CV for both, e.g.:

  • 5-fold within 5-fold:
    5 x (20 test/final validation + 80 optimized training)
    = 5 x (20 test/final validation + 5 x (16 validation/optimization + 64 training))
  • 4-fold within 5-fold:
    5 x (20 test/final validation + 80 optimized training)
    = 5 x (20 test/final validation + 4 x (20 validation/optimization + 60 training)

There's no special reason why the optimization/validation and test/final validation sets need to have the same number of cases. Comparison (particularly, multiple comparison) may require huge numbers of test cases in order to achieve a testing precision that allows meaningful model selection. It then may sound weird to have fewer test cases for the validation of the finally chosen model.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.