Why the data should be resampled under null hypothesis in bootstrap hypothesis testing?

Question

The straightforward application of bootstrap methods to hypothesis testing is to estimate the confidence interval of the test statistic $\hat{\theta}$ by repeatedly calculating it on the bootstrapped samples (Let the statistic $\hat{\theta}$ sampled from bootstrap be called $\hat{\theta^*}$). We reject $H_0$ if the hypothesized parameter $\theta_0$ (which usually equals 0) lies outside of the confidence interval of $\hat{\theta^*}$.

I've read, that this method lacks some power. In the article by Hall P. and Wilson S.R. "Two Guidelines for Bootstrap Hypothesis Testing" (1992) it is written as the first guideline, that one should resample $\hat{\theta^*} - \hat{\theta}$, not the $\hat{\theta^*} - \theta_0$. And this is the part I don't understand.

Isn't that the $\hat{\theta^*} - \hat{\theta}$ measures just the bias of the estimator $\hat{\theta^*}$? For unbiased estimators the confidence intervals of this expression should always be smaller than $\hat{\theta^*} - \theta_0$, but I fail to see, what it has to do with testing for $\hat{\theta}=\theta_0$? There is nowhere I can see we put information about the $\theta_0$.

---

For those of you, who do not have access to this article, this is a quote of the relevant paragraph which comes immediately after the thesis:

To appreciate why this is important, observe that the test will involve rejecting $H_0$ if in $\left| \hat{\theta} - \theta_0\right|$ is "too large." If $\theta_0$ is a long way from true value of $\theta$ (i.e., if $H_0$ is grossly the error) then the difference $\left|\hat{\theta} - \theta_0 \right|$ will never look very much too big compared to the nonparametric bootstrap distribution of $\left| \hat{\theta} - \theta_0\right|$. A more meaningful comparison is with the distribution of $\left| \hat{\theta^*} - \hat{\theta}\right|$. In fact, if the true value of $\theta$ is $\theta_1$ then the power of the bootstrap test increases to 1 as $\left|\theta_1 - \theta_0\right|$ increases, provided test is based on resampling $\left| \hat{\theta^*} - \hat{\theta}\right|$ , but the power decreases to at most the significance level (as $\left|\theta_1 - \theta_0\right|$ increases) if the test is based on resampling $\left|\hat{\theta} - \theta_0\right|$

Answer 1

This is the bootstrap analogy principle. The (unknown) underlying true distribution $F$ produced a sample at hand $x_1, \ldots, x_n$ with cdf $F_n$, which in turn produced the statistic $\hat\theta=T(F_n)$ for some functional $T(\cdot)$. Your idea of using the bootstrap is to make statements about the sampling distribution based on a known distribution $\tilde F$, where you try to use an identical sampling protocol (which is exactly possible only for i.i.d. data; dependent data always lead to limitations in how accurately one can reproduce the sampling process), and apply the same functional $T(\cdot)$. I demonstrated it in another post with (what I think is) a neat diagram. So the bootstrap analogue of the (sampling + systematic) deviation $\hat\theta - \theta_0$, the quantity of your central interest, is the deviation of the bootstrap replicate $\hat\theta^*$ from what is known to be true for the distribution $\tilde F$, the sampling process you applied, and the functional $T(\cdot)$, i.e. your measure of central tendency is $T(\tilde F)$. If you used the standard nonparametric bootstrap with replacement from the original data, your $\tilde F=F_n$, so your measure of the central tendency has to be $T(F_n) \equiv \hat \theta$ based on the original data.

Besides the translation, there are subtler issues going on with the bootstrap tests which are sometimes difficult to overcome. The distribution of a test statistic under the null may be drastically different from the distribution of the test statistic under the alternative (e.g., in tests on the boundary of the parameter space which fail with the bootstrap). The simple tests you learn in undergraduate classes like $t$-test are invariant under shift, but thinking, "Heck, I just shift everything" fails once you have to move to the next level of conceptual complexity, the asymptotic $\chi^2$ tests. Think about this: you are testing that $\mu=0$, and your observed $\bar x=0.78$. Then when you construct a $\chi^2$ test $(\bar x-\mu)^2/(s^2/n) \equiv \bar x^2/(s^2/n)$ with the bootstrap analogue $\bar x_*^2/(s_*^2/n)$, then this test has a built-in non-centrality of $n \bar x^2/s^2$ from the outset, instead of being a central test as we would expect it to be. To make the bootstrap test central, you really have to subtract the original estimate.

The $\chi^2$ tests are unavoidable in multivariate contexts, ranging from Pearson $\chi^2$ for contingency tables to Bollen-Stine bootstrap of the test statistic in structural equation models. The concept of shifting the distribution is extremely difficult to define well in these situations... although in case of the tests on the multivariate covariance matrices, this is doable by an appropriate rotation.

Answer 2

OK, I've got it. Thank you, StasK, for such a good answer. I'll keep it accepted for others to learn, but in my particular case I was missing a very simple fact:

The procedure of bootstrap in accordance to Hall&Wilson guidelines for simple one-sampled mean test is this (in R-inspired pseudo code):

1function(data, $\theta_0$ ) {
2 $\hat{\theta} \leftarrow $ t.test(data, mu = $\theta_0$ )$statistic
3 count $\leftarrow 0$
4for(i in 1:1000){
5 bdata $\leftarrow$ sample(data)
6 $\hat{\theta^*} \leftarrow $ t.test(bdata, mu = $\hat{\theta}$ )$statistic
7 if ( $\hat{\theta^*} \le \hat{\theta} $ ) count++
8 }
9 count/1000
10 }

The part I missed was that the $\theta_0$ was "used" in line 2 (where we set the reference $\hat{\theta}$).

It is interesting to note, that in the line 2 and 6 we could equally easily use p.value instead of statistic. In that case we should also change the $\le$ into $\ge$ in line 7.

Answer 3

Based on the paper and hall-pass's comment, this is how I understood the code in R..

I'm not completely confident I have this correct.

time = c(14,18,11,13,18,17,21,9,16,17,14,15,
         12,12,14,13,6,18,14,16,10,7,15,10)
group=c(rep(1:2, each=12))
sleep = data.frame(time, group)

Mean= tapply(X=sleep$time, INDEX=sleep$group, mean)
t0 = (Mean[1] - Mean[2]) / (sqrt(var(subset(sleep, group == 1)$time) / nrow(subset(sleep, group == 1))))

require(boot)

diff = function(d1,i){
  
  d = d1[i,]
  (mean(d$time) - Mean[1]) / (sqrt(var(d$time) / nrow(d)))
}

set.seed(1234)
b3 = boot(data = subset(sleep, group == 1), statistic = diff, R = 10000)

pvalue = mean(b3$t > t0)
pvalue