I'm reading an article in machine learning and healthcare. This author has a total of 1500 patients, but used only 36 patients for test. He did use 5-fold cross validation for classification using KNN and SVM. I think the test set is too low. Or is it okay to have such low test set - only 2.5%?

  • 2
    $\begingroup$ Can you please provide a link, or at leas the title of the paper? Yes, at first instance the test-set seems indeed small. $\endgroup$
    – usεr11852
    Commented Feb 28, 2018 at 22:29

3 Answers 3


Generally speaking, the more data you use to train your model, the better the predictive power - the more data you hold-out to later test your model, the better the performance estimation.

Since you have tagged "classification" in healthcare, let's just assume a binary outcome ("sick" and "not sick"). A test set containing only 2 sick/34 not sick patients will not be a good estimation.

Patients in the test set need to be representative of all possible patients (feature space). If the author has only used very few features to build his model, maybe the test set is already large enough.

The cross-validation part does not change the fact that the test set might be too small. Data used for validation (to find a good value for k in k-NN) can be regarded as training set and not testing.

  • $\begingroup$ Very good answer. Thank you clearly explaining all parts. This was very helpful. $\endgroup$
    – Kuni
    Commented Feb 28, 2018 at 23:11

Short answer: 36 test patients in a classification setting are typically too few to allow meaningful conclusions from the observed performance.

The size of the test influences uncertainty on the test (i.e. measurement of trained classifier's performance): specifically, the variance (random) uncertainty on the performance estimate. In order to not be biased, you need the test set to be representative for the intended patient population in addition.

This random uncertainty due to the limited test sample size depends on the absolute number of test cases, not on the fraction of the total data.

Assuming the model predicts labels, so predictions can either be correct or wrong, you can describe the testing process as Bernoulli trial where you estimate the probability of a correct prediction and construct confidence intervals.

Here are point estimates and 95% confidence intervals for the situation with 36 test cases: confidence intervals for 36 test cases

As you can see, the confidence interval width ranges from about 10 % (for 0 % or 100 % observed accuracy) to more than 30 % for observed 50:50 guessing.

However, the situation is likely even worse as the overall accuracy is not meaningful unless the proper population prevalence or incedence is reflected in the test set. Then, all that can be meaningfully obtained are sensitivity and specificity (this would be part of the representativity considerations), which are observed fractions on a subset of the test cases. I.e. their confidence intervals are even wider.

If you want to know more details, have a look at our paper:
Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33.
DOI: 10.1016/j.aca.2012.11.007

accepted manuscript on arXiv: 1211.1323


The size of test set is not about % (it could be even 0.5%), but about the number of cases and how representative they are. If the test set is relatively small, but it meaningfully represents the population you want to generalize on, then it is OK. Obviously, the greater size of the test set, the greater chance of having representative sample in test set (assuming random sampling). Notice that this works both ways around: if you have a huge test set, but it is not representative, then it is useless.

  • $\begingroup$ The empirical distribution of the test set $\hat{p}_{test}(x, y)$ should be close to the true data generating distribution $p(x, y)$, so we can have an accurate estimate of the generalization error. What I don't understand is that although we are fine with a representative test set, we usually opt for the largest training set possible. For example, lets say we have a dataset with 100_000 samples and a collection of 10_000 samples is representative of the population. Option 1). 10:90 train:test ratio. Option 2). 10:90 train test ratio. $\endgroup$
    – ado sar
    Commented May 21 at 20:50
  • $\begingroup$ In either case, the training set would be representative of the population. Why most of the time we go with option 1 rather than option 2? $\endgroup$
    – ado sar
    Commented May 21 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.