I am facing the evaluation of two text classifiers. I have a large training dataset (to be used for training only), and a separated small test set (to be used for testing only), both being balanced. Which one of the following methods should be most appropiated one and why?

1) Stratified Repeated held-out evaluation (repeated subsampling): Sample k times without repetition from the training dataset, each sample being balanced. For every sample, classifier is trained with the sample, and accuracy is tested with the full test dataset. Results are averaged.

2) Stratified Cross-validation: Divide the full training dataset in k slices with equal size, each slice being balanced. For every slice, classifier is trained with the slice, and accuracy is tested with the full test dataset. Results are averaged.

3) Stratified Bootstrapping: Sample k times with repetition from the large training dataset, each sample being balanced. For every sample, classifier is trained with the sample, and accuracy is tested with the full test dataset. Results are averaged.


2 Answers 2


I would suggest performing repeated k-fold cross-validation on your training set in order to perform feature and model selection. For each repetition, this would involve randomly partitioning the training set into training (say 90% of original training set) and validation (e.g. 10% of original training set) sets and performing cross-validation.

You can then apply the selected model to your test set to obtain an independent, unbiased estimate of your classifier's generalized performance.

Edits based on comments below:

  1. The purpose of the 10 fold CV is to validate the procedure you use to train your model, and also to obtain an unbiased performance estimate. You can get your final trained model by applying all training set data to the feature and model selection procedures once you are happy with the model estimate produced through CV on the training data set.
  2. You should only test your model using the test set once using all the available test set data
  3. I have never used bootstrapping and so can't comment other than that I believe it to be an equally valid approach
  • $\begingroup$ Suppose Stratified 10-fold cross validation. 3 questions: (1) The purpose of the validation set is just selecting the best model between the 10 models obtained ? (1 for each iteration), (2) In this case, the selected model should be tested only once in the end with the whole test set? (3) What are the benefits of this compared to bootstrapping (in this similar question, bootstrapping seems to be recommended) $\endgroup$
    – SUP
    Oct 31, 2012 at 1:33
  • $\begingroup$ Added some edits in response to your questions above $\endgroup$
    – BGreene
    Oct 31, 2012 at 8:48
  • $\begingroup$ +1 for the important point of performing the model/feature selection independently in each fold of the performance evaluation procedure. If you don't do that correctly, the differences between the three resampling approaches are irrelevant! $\endgroup$ Oct 31, 2012 at 9:25
  • $\begingroup$ Thanks for your response. One last question and I mark this as the answer. Suppose that I don't want to use any data from the training dataset to validate the classifier. Would it be possible to perform CV with only separated training and test sets? If possible, how? $\endgroup$
    – SUP
    Oct 31, 2012 at 10:04
  • $\begingroup$ Well, you would then need to be able to partition both test and training sets to create different test/train pairs in order to perform a cross-validation. As @DikranMarsupial states just doing one repetition of training on the training set and testing with test set has the potential for high variance. This could be a suitable scenario for bootstrapping $\endgroup$
    – BGreene
    Oct 31, 2012 at 10:06

The problem here is that the test set is small, so even if the performance estimate from the hold out set is unbiased, it will have a high variance, and hence be unreliable (to some degree). Repeated hold out, cross-validation or bootstrapping are all good methods to reduce the variance of the estimate, and I suspect there isn't a great deal to choose between them (I tend to use repeated hold-out and bootstrapping a fair bit - repeated hold-out when I just wan't a performance estimate and bootstrapping when I want to make a bagged model as an extra guard against over-fitting).

The real issue is that the estimated variance of all three approaches will be optimistically biased as the estimates from each iteration of the re-sampling will not be independent. So take the confidence interval on the mean with a pinch of salt.

There is a nice book on "Evaluating Learning Algorithms - A Classification Perspective" by Japkowicz and Shah, I've only skimmed it so far, but I think it is a book that ought to be on the shelves of machine learning researchers and practitioners everywhere.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.