1. When the sample is small and study is under-powered, can bootstrapping be used to generate data and increase the sample size (and improve the test power)?
  2. Is there any way that bootstrapping can be used to improve the power, without generating data?
  3. When bootstrapping is used to generate data (if possible), must the level of significance be reduced to compensate for the inflated power due to the fake newly generated data?
  4. Can you cite any references about "bootstrapping to improve power, and its rules"?

This is a general question. My concern is not just to increase the power (by any methods including using a better test or increasing the sample size), but increase the power via bootstrapping. I just want to know (by some references) whether bootstrapping can be a remedy for small, under-powered samples? And if so, how?


closed as too broad by kjetil b halvorsen, mdewey, usεr11852, Michael Chernick, gung Mar 21 at 13:24

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 5
    $\begingroup$ You can always improve the power without generating data: use a decision rule that invariably rejects the null hypothesis. The problem with your question is that the power depends on the statistical test as well as the model of the data, and both of those will determine to what extent the power might be improved by selecting some alternative procedure (whether bootstrapping or something else). Could you therefore edit this question to make it a little less general and more focused on the problem you actually face? $\endgroup$ – whuber Mar 20 at 12:13
  • $\begingroup$ Thanks dear whuber. My concern was not to increase the power of my study, but to check, as a general question, to see if bootstrapping can be used in any way to increase the power. Lets say there is a study design where the best test is being used for analyzing a particular model. Yet the test is still under-powered, because the study is a pilot one, it is not possible to add more datapoints. Now, can bootstrapping be used to increase the power? $\endgroup$ – Vic Mar 21 at 5:33
  • 1
    $\begingroup$ By definition and assumption, that is not possible: when you apply the best test--in the sense of being most powerful for its size--it's not possible to do better. $\endgroup$ – whuber Mar 21 at 13:32
  • $\begingroup$ That seems what I wanted. Whuber I see my question is suspended. Please note that it is a specific question (with 2 branches). It is not about a single study. It is a good question never asked before on the internet (to my knowledge), so let it live please. $\endgroup$ – Vic Mar 22 at 4:42
  • 1
    $\begingroup$ That shouldn't be the case, Vic: since you are the originator, you have the ability to edit the post. I am reluctant unilaterally to override the votes of five other users: the fact that they voted to close this question means there is a genuine need to clarify it. $\endgroup$ – whuber Mar 22 at 12:47

That may be possible in some cases, although you should not expect miracles.

The main reason to use a bootstrap for testing purposes (which is how I read your reference to "power") is usually to find an approximation to the null distribution of test statistics.

Here is a simple example. I generate data x from (of course, that is something which is unknown in practice!) a N(3,1), where it is assumed (just for simplicity) that we know the variance is one.

x.star generates, by sampling with replacement from x, resamples, B times in total. I.e., we then sample from a sample with mean xbar. In order to get bootstrap test statistics that behave like standard normals - i.e., the behavior of the test statistic when the null is true - we therefore need to subtract the "bootstrap population mean" xbar from each of the sample averages of the bootstrap samples mean(x.star). This is analogous to subtracting the mean under null, $\mu_0$ from $\bar x$ in the numerator of a standard non-bootstrap test statistic.

When you do that many, B, times, we can use these bootstrap test statistics to build a bootstrap distribution that may serve as a null distribution against which to compare the realized test statistic:

n <- 5000
x <- rnorm(n, 3)     # data
xbar <- mean(x)      # mean
B <- 1e5             # number of bootstrap resamples
t.star <- rep(NA,B)

for (b in 1:B){
  x.star <- sample(x, n, replace=T)             # bootstrap data
  t.star[b] <- sqrt(n)*(mean(x.star)-xbar)      # bootstrap test statistic - by subtracting xbar, we make it behave like a N(0,1)

t <- abs(sqrt(n)*xbar)                          # test statistic for testing mu=0

bs.cv <- quantile(abs(t.star), .95)             # bootstrap critical value. reject if abs(t) > bs.cv
qnorm(.975)                                     # asymptotic c.v. For n,B large, bs.cv \approx bs.cv

> bs.cv
> qnorm(.975)
[1] 1.959964

Of course, in practice we would rather use such approaches where the usual asymptotic approximation may be untractable, e.g., due to nuisance parameters whose effect cannot (easily) be accounted for.

That said, there is much literature on bootstrap tests providing asymptotic refinements (e.g., here), i.e., faster convergence of the rejection rates to the nominal level. If the error-in-rejection probability of the underlying test using standard asymptotic critical values was such that it underrejected, a refinement will more completely exhaust the nominal rejection probability, and therefore, typically, also higher power, as a test that rejects more frequently when the null is true typically also does so when it is not.

  • 3
    $\begingroup$ I believe the bootstrap is not usually justified as you claim. The point is that the data you have are supposed to reflect the population. If the null does not hold, then doesn't it clearly follow that sampling from the data will not sample from the null? $\endgroup$ – whuber Mar 21 at 13:33
  • 1
    $\begingroup$ @whuber, of course, when you resample from the sample directly (and I certainly did not write that this is all one should do!), that will not approximate the null distribution, which is why bootstrapping the null distribution of hypothesis tests involves a step where you ensure that the behaviour of the bootstrap test statistics mimics that of the test statistic when the null is true. The actual test statistic, in turn, of course won't be from that distribution and hence be larger, which is where the bootstrap test gets its power from. $\endgroup$ – Christoph Hanck Mar 21 at 17:41
  • 1
    $\begingroup$ Please refer to the above illustration/edit and, e.g., Horowitz, Handbook of econometrics, Ch. 52, Sec. 3.3 for more discussion. So, I stand by my answer. $\endgroup$ – Christoph Hanck Mar 21 at 17:41
  • 2
    $\begingroup$ "When you resample from the sample directly" is practically the definition of the bootstrap. Perhaps you have in mind some kind of parametric bootstrap? $\endgroup$ – whuber Mar 21 at 17:45
  • 1
    $\begingroup$ Yes, that is kind of the definition of the bootstrap. The rest of my answer tries to indicate how to use that bootstrap sample for the purposes of hypothesis testing via building a null distribution that may serve as a (possibly superior) alternative to the more conventional asymptotic null distribution. My point is unrelated to parametric vs. nonparametric bootstraps. $\endgroup$ – Christoph Hanck Mar 21 at 17:50

I like Christoph's answer (+1) but I think your question reflects some basic misunderstanding of what bootstrap is and how it works. It's true that bootstrapping generates data, but this data is used to get a better idea of the sampling distribution of some statistic, not to increase power Christoph points out a way that this may increase power anyway, but it's not by increasing the sample size.

If you could do that, then everyone would use very small samples and just bootstrap bigger ones. It doesn't work that way.

  • 1
    $\begingroup$ Thanks dear Peter. This is exactly what I wanted to hear; this is why I called the newly generated data [if added to the original sample] as a fake one; yet I have colleagues who do that! But do you have a text-book sort of reference for it? $\endgroup$ – Vic Mar 22 at 4:52
  • 2
    $\begingroup$ I don't know of a textbook that explicitly says this, but any book on bootstrapping implies it. The generated data is in a separate universe. $\endgroup$ – Peter Flom Mar 22 at 13:13

Not the answer you're looking for? Browse other questions tagged or ask your own question.