Bootstrapping test set? Let's say I have a classification problem with a small and fixed test set.
If I train a classifier and report the accuracy on this test set, I know that this estimate has a high variance. Does it make sense to bootstrap the test set to reduce the variance of the accuracy estimate? 
 A: I can't imagine how manipulating with your test set could lead to anything more then cheating yourself that your results are better then in reality.
The idea of cross-validation and testing your results on unseen data, is to approximate the situation where your model would be applied to some future, unknown data. This approximation may be better or worse depending on how similar is the data you have to the future data.
The idea of bootstrap is that you sample from your data the same way as you'd sample from the population, so to approximate the sampling process and estimate the variability caused by it. First thing to notice is that such procedure does not let you learn anything about possible performance if the data you have is not similar to the future data. Second, the sampling is ought to imitate the sampling process, so rather then resampling your test set, you should instead make multiple random splits to train and test set (i.e. use k-fold cross-validation).
Finally, bootstrap is designed for estimating the possible variability, not for correcting it. Bradley Efron himself discouraged from such useage of bootstrap. The completely different story, is to bootstrap resample the train set and then aggregate the results, i.e. use bagging -- this would help to reduce the variance of the predictions.
A: I assume that you also have a (bigger) training set, and that the training and test set have the same relation between features and target variable (there is no significant difference). Then: no, bootstrapping your test set and considering the performance over the resulting test sets is most likely not helpful. 
You use your training data to e.g. select features, train, and evaluate different model types and hyperparameters. From those results you chose one "best suited" model for the job. This is the one you should evaluate on your test set once for reassurance that everything is OK. With this setup, splitting the test set does not bring benefits anymore: if would not reuse samples during bootstrapping, you would just get "subresults", which then would be averaged to one scalar result (and cause less granularity thereby), or you would bootstrap with replacement, which with few partitions would cause better or worse results by chance (and on an infinite amount of rounds would give you very similar results again).
If you are really stuck with a very small test set, and test performance is the only thing that counts (why would it be? You aim for generalization, right?), the question boils down to how well test data represents the real application case data - because few samples might just be too less to allow for any good estimate of real world performance. If you think this might be true for your case, getting/asking for more test data might be required anyway.
A: You are trying to estimate the MSE. You have "n" points, all of which are an unbiased estimate of the MSE. Even if you keep resampling and averaging, you cannot get a more efficient estimator of your MSE than what you already have from the test set.
This is analogous to drawing 10 samples and estimating the population mean from the sample average - you cannot make a more (unbiased) efficient estimator than that one.
