I am working my way through Chapter 4 (Tests) in Davison & Hinkley's (D&H's) book "Bootstrap Methods and their Applications", and have a question about one of their examples.

Davison & Hinkley's proposed test

Example 4.1 proposes a permutation test for the (alternative) hypothesis of non-zero slope in a simple logistic regression. Quoting from page 141:

Example 4.1 (Logistic regression) Suppose that $y_1, \ldots , y_n$ are independent binary outcomes, with corresponding scalar covariate values $x_1, \ldots , x_n$ and that we wish to test whether or not $x$ influences $y$. If our chosen model is the logistic regression model $$ \log \frac{\Pr(Y_j = 1 | x_j)}{\Pr(Y_j = 0 | x_j} = \lambda + \psi x_j, \quad j=1,\ldots,n, $$ then the null hypothesis is $H_0 : \psi=0.$ Under $H_0$ the sufficient statistic for $\lambda$ is $S=\sum Y_j$ and $T=\sum x_j Y_j$ is the natural test statistic

[...snipped text...]

The null distribution of $Y_1,\ldots,Y_n$ given $S = s$ is uniform over $n \choose s$ permutations of $y_1, \ldots, y_n$.

Rather than compute all of those permutations to get the exact distribution of $T$, they propose leaving the $x_i$ fixed and generating $R$ permutations of the $y_i$ to approximate $T$'s exact distribution.

My naïve test

That all seems reasonable, but my thought would have been to simulate a slightly different null distribution. Given that $\psi = 0$ under the null hypothesis, $$ \Pr(Y_j = 1| x_j) = \frac{e^{\lambda}}{e^{\lambda} + 1}, \quad j=1,\ldots,n, $$

Leaving $x_1, \ldots , x_n$ fixed, I would produce $R$ sets of $y^*_1, \ldots , y^*_n$ with each $y^*_i \sim \textrm{Bernoulli}(\frac{e^{\lambda}}{e^{\lambda} + 1})$. Instead of (but probably equivalently to) D&H's $T=\sum x_j Y_j$, I would take as my test statistic the estimate of $\psi$ from a fitted logistic regression of the bootstrap sample.


The two tests above depend on different distributions of bootstrap samples. Mine, for instance, may produce some samples in which all of the $y^*_i$ equal 1, whereas all of D&H's samples contain the same number of 1's. The salient difference seems to be that D&H's sampling distribution is conditional on $S=\sum Y_j$ whereas mine is not.

Is one of these approaches preferable to the other in all situations? They seem to assume/model different data-generation processes in the originally sampled populations. Is that correct? If so, when should I choose one approach over the other?


1 Answer 1


Davison & Hinkley, I now believe, would argue that, for this problem, their proposed method is preferable.

Example 4.1 comes near the start of a chapter in which the authors first develop a big-picture overview of several approaches to hypothesis testing and only then introduce bootstrap hypothesis tests. In discussing hypothesis tests for which the null hypothesis $H_0$ is composite, they lay out several approaches to forming a test:

  1. Very rarely, one may carefully choose a test statistic $T$ with distribution that is the same for all $F$ satisfying $H_0$. A familiar example is the Student's $t$-test for a normal mean with unknown variance.

  2. Failing that, and more often, one may be able to eliminate the parameters that remain unknown when $H_0$ is true by conditioning on the sufficient statistic (denoted by $S$) under $H_0$. This produces a conditional $P$-value defined by:

    $$ p = \mathrm{Pr}(T \geq t | S=s, H_0) $$

    Permutation tests, which condition on the marginal EDFs as their sufficient statistic, are good examples of this approach.

  3. Where that is not possible, "a less satisfactory approach, which can nevertheless give good approximations is to estimate $F$ by a CDF $\hat{F}_0$ which satisfies $H_0$ and then calculate":

    $$ p = \mathrm{Pr}(T \geq t | \hat{F}_0) $$

    This is the approach taken by bootstrap hypothesis tests.

Using the terminology that Davison & Hinkley use, their Example 4.1 is a conditional test that implements the second approach given above. My proposed solution is a bootstrap test that follows the logic in the third approach given above.

Edit of 2017-02-22

Romano (1989), mentioned in the Bibliographic Notes of D&H's Chapter 4, elucidates and explores in much greater detail the distinction between permutation (conditional) tests and bootstraps tests.


Romano, J. P. Bootstrap and randomization tests of some nonparametric hypotheses. 1989. The Annals of Statistics. 17(1), 141–159. http://dx.doi.org/10.1214/aos/1176347007


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.