# Bootstrapping power estimates for a bootstrap test

Assume I want to use a (nonparametric) bootstrap test for a hypothesis with a sample size of $n_1$ and I already have $n_0$ actual samples on which to base my power estimates. Usually, we would also have $n_1 > n_0$.

Is it a valid procedure to estimate the power of my test by using nested bootstrapping? Basically, I would repeatedly sample $n_1$ samples with replacement from my $n_0$ samples and apply my non-parametric bootstrap test to each of these samples. Finally, I would look at the percentage of the bootstrap tests which was significant at my $\alpha$ level. Are there pitfalls I have to look out for?

I already did some googling without much success, perhaps because I do not know the proper search terms. Therefore it would be also nice to know how the procedure (if it exists) is called, so I can find references.

• So if I've got you correctly, you've got some sample from a larger population. You're thinking of using that small sample to make lots of different samples (using some resampling procedure), and then applying your test (which happens to involve bootstrapping) to see how often your test is successful, getting power. Is that right? If so, it would seem to me that the validity would depend quite a lot on how well the small sample approximates the big-sample-to-be-collected. Commented May 18, 2013 at 5:40
• Commented May 18, 2013 at 5:45
• The technique you describe seems to be a fusion of the double bootstrap and the m-out-of-n bootstrap. This seems to be slightly related but doesn't cover the m-out-of-n aspect and I have only read its abstract. Commented Aug 19, 2013 at 4:45
• Also your application of the m-out-of-n bootstrap is non-standard (usually it is used to restore consistency not to simulate a larger sample size) and I am unsure if it will work as you envisage. Commented Aug 19, 2013 at 4:51

As you bootstrap you assume that the new bootstrapped distribution is equivalent to the original distribution. If $n_1>n_0$ then you are forced to draw repeated values of $n_0$, which leads to several problems. First of all, the bootstrapped distribution will only have de facto $n_0$ values, the rest being copies, and these copies will only lead to a stronger weighting of these -- randomly drawn -- value when doing e.g. a correlation analysis. Then there is the issue on how your tests might require non-correlated data, as referred by @nic.
If you want to really attain $n_1$ real independent values, I think you have to analyse the $n_0$ values, understand which distribution they are likely to follow and then draw $n_1$ values from this assumed distribution. It is certainly not as generic, but, IMO, is more transparent and you will not suffer from the issues mentioned above.