Assume I want to use a (nonparametric) bootstrap test for a hypothesis with a sample size of $n_1$ and I already have $n_0$ actual samples on which to base my power estimates. Usually, we would also have $n_1 > n_0$.

Is it a valid procedure to estimate the power of my test by using nested bootstrapping? Basically, I would repeatedly sample $n_1$ samples with replacement from my $n_0$ samples and apply my non-parametric bootstrap test to each of these samples. Finally, I would look at the percentage of the bootstrap tests which was significant at my $\alpha$ level. Are there pitfalls I have to look out for?

I already did some googling without much success, perhaps because I do not know the proper search terms. Therefore it would be also nice to know how the procedure (if it exists) is called, so I can find references.

  • $\begingroup$ So if I've got you correctly, you've got some sample from a larger population. You're thinking of using that small sample to make lots of different samples (using some resampling procedure), and then applying your test (which happens to involve bootstrapping) to see how often your test is successful, getting power. Is that right? If so, it would seem to me that the validity would depend quite a lot on how well the small sample approximates the big-sample-to-be-collected. $\endgroup$ Commented May 18, 2013 at 5:40
  • 1
    $\begingroup$ Also: cdn.memegenerator.net/instances/400x/37929308.jpg $\endgroup$ Commented May 18, 2013 at 5:45
  • $\begingroup$ The technique you describe seems to be a fusion of the double bootstrap and the m-out-of-n bootstrap. This seems to be slightly related but doesn't cover the m-out-of-n aspect and I have only read its abstract. $\endgroup$
    – orizon
    Commented Aug 19, 2013 at 4:45
  • $\begingroup$ Also your application of the m-out-of-n bootstrap is non-standard (usually it is used to restore consistency not to simulate a larger sample size) and I am unsure if it will work as you envisage. $\endgroup$
    – orizon
    Commented Aug 19, 2013 at 4:51

2 Answers 2


I don't think use bootstrap to artificially increase your sample size would be a good idea. Any violation of the assumption of the independence of the observations dramatically increase the odds od a spurious result (which would be the case when n1 is significantly greater than n0).

I would estimate the confidence interval of the effect size (the strength of the difference / relationship you are trying to use) and assume the lower bound as the true effect. Then it would be easy to estimate the power.

[note: I assume you already have a significan result with n0 observations. Otherwise your data is compatible* with the null hypothesis and there is no way to have a conservative estimate of the powere - unless you are using the wrong test. Power analysis assume the knowledge of the "real" effect size so there is no way to use theme to "bypass" inferential statistics]

*likely to be observed if the null hypotesis is true


As you bootstrap you assume that the new bootstrapped distribution is equivalent to the original distribution. If $n_1>n_0$ then you are forced to draw repeated values of $n_0$, which leads to several problems. First of all, the bootstrapped distribution will only have de facto $n_0$ values, the rest being copies, and these copies will only lead to a stronger weighting of these -- randomly drawn -- value when doing e.g. a correlation analysis. Then there is the issue on how your tests might require non-correlated data, as referred by @nic.

If you want to really attain $n_1$ real independent values, I think you have to analyse the $n_0$ values, understand which distribution they are likely to follow and then draw $n_1$ values from this assumed distribution. It is certainly not as generic, but, IMO, is more transparent and you will not suffer from the issues mentioned above.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.