# How to choose the training, cross-validation, and test set sizes for small sample-size data?

Assume I have a small sample size, e.g. N=100, and two classes. How should I choose the training, cross-validation, and test set sizes for machine learning?

I would intuitively pick

• Training set size as 50
• Cross validation set size 25, and
• Test size as 25.

But probably this makes more or less sense. How should I really decide these values? May I try different options (though I guess it is not so preferable... increased possibility of over learning)?

What if I had more than two classes?

• 100 is too small for me. I would opt for a leave-one-out strategy for both cross-validation and test evaluation. – Memming Sep 1 '14 at 18:55
• I haven't seen any literature on this (minimum sample sizes for validation). Not sure why. Seems like an important issue. – charles Sep 1 '14 at 20:17

• You surely found the very similar question: Choice of K in K-fold cross-validation ?
(Including the link to Ron Kohavi's work)

• If your sample size is already small I recommend avoiding any data driven optimization. Instead, restrict yourself to models where you can fix hyperparameters by your knowledge about model and application/data. This makes one of the validation/test levels unnecessary, leaving more of your few cases for training of the surrogate models in the remaining cross validation.
IMHO, you anyways cannot afford very fancy models with that sample size. And almost certainly you cannot afford to do any meaningful model comparisons (for sure not unless you use proper scoring rules and paired analysis techniques).
This decision is far more important than the precise choice of $k$ (say, 5-fold vs. 10-fold) - with the important exception that leave one out is not recommended in general.

• Interestingly, with these very-small-sample-size classification problems, validation is often more difficult (in terms of sample size needs) compared to training of a decent model. If you need any literature on this, see e.g. our paper on sample size planning:
Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33.
DOI: 10.1016/j.aca.2012.11.007

accepted manuscript on arXiv: 1211.1323

• Another important point is to make good use of the possibility to iterate/repeat the cross validation (which is one of the reasons against LOO): this allows you to measure the stability of the predictions against perturbations (i.e. few different cases) of the training data.

Literature:

• If you decide for a single run on a hold-out test set (no iterations/repetitions),

• +1 purely for the advice on parameter optimization and model complexity. but all of this advice is fantastic. – charles Sep 3 '14 at 15:46

Given that your sample size is small a good practice would be to leave out the cross-validation section and use a 60 - 40 or 70 - 30 ratio.

As you can see in section 2.8 of Introduction to Clementine and Data Mining and also in MSDN Library - Data Mining - Training and Testing Sets a 70 - 30 ratio is common. According to Andrew Ng's Machine Learning lectures a 60 - 20 - 20 ratio is recommended.

Hope I was helpful. Best Regards.