How to choose the test set size when the training set size is given? I have data on 64 subjects collected in a medical setting. 
With the help of ROC curve analysis and bootstrapping, I have identificed two predictors for illness(present or not present) in the group. 
I now want to test my model on a new data set, but collecting new data is time consuming and expensive. 
How do I determine the size of my testing set?
 A: (Hello and welcome to cross validated).
Assuming that you are supposed to report test results as sensitivity and specificity and/or positive and negative predictive values: these are observed proportions of particular subsets of your test data. 
You can calculate necessary test sample size, i.e. the number of cases needed in the denominator of the respective fractions for several scenarios. 


*

*The easiest and maybe most straightforward possibility is to specify what an acceptable uncertainty (or precision) in the test result is, e.g. sensitivity should have a 95% confidence interval of < 0.1 width (≈ p ± 5%). 

*A second scenario is that you need to show that your model has at least a given performance, and you have a guesstimate (e.g. by cross validation) of its performance (which is better than required). You can then calculate from the required performance and assuming the observed performance is the true performance of the model how many cases you need in order to have the one-tailed confidence interval ending at the required performance.


We wrote a paper explaining these strategies: Beleites, C. and Neugebauer, U. and Bocklitz, T. and Krafft, C. and Popp, J.: Sample size planning for classification models. Anal Chim Acta, 2013, 760, 25-33.
DOI: 10.1016/j.aca.2012.11.007
accepted manuscript on arXiv: 1211.1323

I greatly appreciate that you plan for a proper validation study. This is far too often neglected. However, in order to make sure that the experimental effort is not wasted I strongly recommend to consult your local (medical) statistician about the design of the validation experiment. There's far more to planning a good experiment than just getting the sample size right!
