Suppose we have data $X_1,\ldots, X_n$ and we want to do some Hypothesis test based on the value of a statistic $T.$ Let's say that the larger the value of $T,$ the more "likely" the hypothesis is rejected. It is customary to do the following:

  1. Draw $B$ sets of bootstrap samples from $X_j.$
  2. Compute the value of $T(X_j^b)$ for each bootstrap sample $b=1\ldots B.$
  3. Use the rank of $T(X_j)$ among all values of $T$ evaluated to determine whether to accept or reject the hypothesis.

(Or a modified version of this if the classic version somehow does not work.)

In some books I have read, the explanation terminates at "the test (clearly) has the desired size". But what's more important is the power of the test. The test should be more likely to reject $H_0$ when $H_0$ is not true. And clearly it does not work in some situations. For example, if $T$ is the sample mean and $X_j$ are iid from $N(\theta,1),$, and $H_0:\theta = \theta_0, H_1:\theta>\theta_0,$ then this test will be almost completely useless.

The hope is that the bootstrap example should emulate the true distribution, but this is not immediately clear for two reasons:

  1. The sample itself is already deviated from the true distribution, so the bootstrap is biased (as shown in the example above).
  2. We actually do not need to emulate the true distribution fully. All we need is to make the test work, and this is not mathematically equivalent to fully accurate emulations.

The question is: In which situation is the test powerful, and in which situation the test has low power?

Are there any general theorems proven on these things? Or is there an intuitive explanation which paints the whole big picture?

  • $\begingroup$ You need to ensure that the null hypothesis is true for the bootstrap "population". You can do this by adjusting the raw values that you resample. $\endgroup$ Aug 30, 2023 at 10:32
  • $\begingroup$ @BigBendRegion What do you mean by "adjust"? $\endgroup$
    – Ma Joad
    Aug 30, 2023 at 13:09
  • $\begingroup$ How do you conclude the test is "almost completely useless" for a Normal distribution? It works beautifully when bootstrapping is properly applied, suggesting you might have something else in mind. Indeed, what exactly do you mean by the "true" distribution"? Are you perhaps confusing that with the hypothetical distribution implied by the null hypothesis? $\endgroup$
    – whuber
    Aug 30, 2023 at 14:37
  • $\begingroup$ @whuber By "completely useless" I mean the particular situation I have described. You can always make it work by using a different setup, but that is not the point. $\endgroup$
    – Ma Joad
    Aug 31, 2023 at 2:30
  • $\begingroup$ @whuber By true distribution I mean the actual distribution of the random variable studied, no matter it satisfies the null hypothesis or not. $\endgroup$
    – Ma Joad
    Aug 31, 2023 at 2:31


This site is temporarily in read-only mode and not accepting new answers.

Browse other questions tagged .