Numerical leave-some-out, k-fold cross validation example please

Question

I am the kind that understands much easier on numerical examples.

Assume we will use 5-fold cross-validation. Can you calculate number of examples in training, test and validation sets for a data that have 100 examples in total?

Thanks a lot.

Answer 1

k-fold cross validaton splits into 2 sets, i.e. for 100 cases 5-fold CV

• 5 x (20 validation/optimization + 80 trainig), or
• 5 x (20 test/final validation + 80 optimized training)

Splitting into 3 (training + validation/optimization + test/final validation) sets e.g. by nesting k-fold cross validation. You can choose the inner and outer validation schemes independently, including k-fold CV for both, e.g.:

• 5-fold within 5-fold:
  5 x (20 test/final validation + 80 optimized training)
  = 5 x (20 test/final validation + 5 x (16 validation/optimization + 64 training))
• 4-fold within 5-fold:
  5 x (20 test/final validation + 80 optimized training)
  = 5 x (20 test/final validation + 4 x (20 validation/optimization + 60 training)

There's no special reason why the optimization/validation and test/final validation sets need to have the same number of cases. Comparison (particularly, multiple comparison) may require huge numbers of test cases in order to achieve a testing precision that allows meaningful model selection. It then may sound weird to have fewer test cases for the validation of the finally chosen model.