Evaluation method when using a large training set and a small test set I am facing the evaluation of two text classifiers. I have a large training dataset (to be used for training only), and a separated small test set (to be used for testing only), both being balanced. Which one of the following methods should be most appropiated one and why?
1) Stratified Repeated held-out evaluation (repeated subsampling):
Sample k times without repetition from the training dataset, each sample being balanced. For every sample, classifier is trained with the sample, and accuracy is tested with the full test dataset. Results are averaged. 
2) Stratified Cross-validation:
Divide the full training dataset in k slices with equal size, each slice being balanced. For every slice, classifier is trained with the slice, and accuracy is tested with the full test dataset. Results are averaged. 
3) Stratified Bootstrapping:
Sample k times with repetition from the large training dataset, each sample being balanced. For every sample, classifier is trained with the sample, and accuracy is tested with the full test dataset. Results are averaged.
 A: I would suggest performing repeated k-fold cross-validation on your training set in order to perform feature and model selection. For each repetition, this would involve randomly partitioning the training set into training (say 90% of original training set) and validation (e.g. 10% of original training set) sets and performing cross-validation.
You can then apply the selected model to your test set to obtain an independent, unbiased estimate of your classifier's generalized performance.
Edits based on comments below:


*

*The purpose of the 10 fold CV is to validate the procedure you use
to train your model, and also to obtain an unbiased performance
estimate. You can get your final trained model by applying all training set
data to the feature and model selection procedures once you are happy with the model estimate produced through CV on the training data set.

*You should only test your model using the test set once using all the available test set data

*I have never used bootstrapping and so can't comment other than that I believe it to be an equally valid approach

A: The problem here is that the test set is small, so even if the performance estimate from the hold out set is unbiased, it will have a high variance, and hence be unreliable (to some degree).  Repeated hold out, cross-validation or bootstrapping are all good methods to reduce the variance of the estimate, and I suspect there isn't a great deal to choose between them (I tend to use repeated hold-out and bootstrapping a fair bit - repeated hold-out when I just wan't a performance estimate and bootstrapping when I want to make a bagged model as an extra guard against over-fitting).
The real issue is that the estimated variance of all three approaches will be optimistically biased as the estimates from each iteration of the re-sampling will not be independent.  So take the confidence interval on the mean with a pinch of salt.
There is a nice book on "Evaluating Learning Algorithms - A Classification Perspective" by Japkowicz and Shah, I've only skimmed it so far, but I think it is a book that ought to be on the shelves of machine learning researchers and practitioners everywhere.
