Let's say I have a classification problem with a small and fixed test set. If I train a classifier and report the accuracy on this test set, I know that this estimate has a high variance. Does it make sense to bootstrap the test set to reduce the variance of the accuracy estimate?


I can't imagine how manipulating with your test set could lead to anything more then cheating yourself that your results are better then in reality.

The idea of cross-validation and testing your results on unseen data, is to approximate the situation where your model would be applied to some future, unknown data. This approximation may be better or worse depending on how similar is the data you have to the future data.

The idea of bootstrap is that you sample from your data the same way as you'd sample from the population, so to approximate the sampling process and estimate the variability caused by it. First thing to notice is that such procedure does not let you learn anything about possible performance if the data you have is not similar to the future data. Second, the sampling is ought to imitate the sampling process, so rather then resampling your test set, you should instead make multiple random splits to train and test set (i.e. use k-fold cross-validation).

Finally, bootstrap is designed for estimating the possible variability, not for correcting it. Bradley Efron himself discouraged from such useage of bootstrap. The completely different story, is to bootstrap resample the train set and then aggregate the results, i.e. use bagging -- this would help to reduce the variance of the predictions.

  • $\begingroup$ "Model accuracy is reported on the test set, and 1000 bootstrapped samples were used to calculate 95% confidence intervals. " can you comment on this from arxiv.org/pdf/1801.07860.pdf ? $\endgroup$ – user0 Feb 25 '18 at 21:06
  • $\begingroup$ @user86895 at face value it is ok, but why only the test set is resampled? Why to use single test set? Wouldn't it be better to resample whole data, then randomly split to train and test, estimate and evaluate your model on them? $\endgroup$ – Tim Feb 25 '18 at 21:18
  • 1
    $\begingroup$ Agree --not sure why they did it this way, nested cv or bootstrap as you describe would be preferable. Generally with enormous datasets I have observed that only a test set is taken, but this dataset only has a few hundred thousand cases...especially for very rare diagnoses this could lead to a very weird test sample. Will open a new question $\endgroup$ – user0 Feb 25 '18 at 21:22
  • $\begingroup$ The reason that both the paper you linked as well as a lot of other papers do bootstrapping only on the test set, is merely practical: it can be really time consuming to both train and evaluate a model using a full bootstrapping process (which affects both training and test sets). A deep learning model can easily take 2 days to train, imagine having to retrain 100 times in order to just evaluate one model, and now you have to do this again for each model you want to compare this against. $\endgroup$ – Ataxias Jul 17 '19 at 17:33
  • $\begingroup$ @Ataxias what is the test case that it tests? I can't see how does this mesure anything of practical importance. $\endgroup$ – Tim Jul 17 '19 at 18:14

I assume that you also have a (bigger) training set, and that the training and test set have the same relation between features and target variable (there is no significant difference). Then: no, bootstrapping your test set and considering the performance over the resulting test sets is most likely not helpful.

You use your training data to e.g. select features, train, and evaluate different model types and hyperparameters. From those results you chose one "best suited" model for the job. This is the one you should evaluate on your test set once for reassurance that everything is OK. With this setup, splitting the test set does not bring benefits anymore: if would not reuse samples during bootstrapping, you would just get "subresults", which then would be averaged to one scalar result (and cause less granularity thereby), or you would bootstrap with replacement, which with few partitions would cause better or worse results by chance (and on an infinite amount of rounds would give you very similar results again).

If you are really stuck with a very small test set, and test performance is the only thing that counts (why would it be? You aim for generalization, right?), the question boils down to how well test data represents the real application case data - because few samples might just be too less to allow for any good estimate of real world performance. If you think this might be true for your case, getting/asking for more test data might be required anyway.

  • $\begingroup$ Yes I am aiming for generalization. Other than test performance, what can you look at to gauge generalization? Also, I wasn't saying that bootstrapping the test set will improve your model. I was saying that it may improve the test accuracy estimate. At the very least, you will get a sense of the variance of the estimate, which you cannot get by just evaluating the accuracy once. $\endgroup$ – Jessica Jul 26 '16 at 21:39
  • $\begingroup$ @PyDumb Yes, looking at the test performance is the right thing. I didn't mean that bootstrapping your test set would improve the model. What I meant is that bootstrapping your test set won't give you a better estimate of the true model performance, as you can't introduce new information by reusing samples with one single, final model. Concerning variance: if you do random sampling of your test set and truly only use a final model and classification decision threshold, your variance won't give you additional information either afaiu. $\endgroup$ – geekoverdose Jul 27 '16 at 9:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.