I've been trying to look into the possibility of plucking a p-value for a slightly tricky case from the bootstrap distribution that I'm generating to construct confidence intervals. Everything I'm reading, including here on StackExchange (e.g. https://stats.stackexchange.com/a/277391/54668) and elsewhere, talks about rejigging the data so that the bootstrap samples represent the distribution of the statistic under the null. This makes sense as a way forward. But what I don't understand is, why can we not just look at the percentile of the null hypothesis parameter value in the bootstrap samples we used to generate our CI? I know that is not what the bootstrap samples are supposed to model, but my thinking is as follows:

I understand that the sampling distribution (if the alternative is true) can be wildly different from that if the null is true and further that the bootstrap samples model the sampling distribution of the estimate... But... the following logic then leaves me confused. For simplicity of explanation, I'll talk about one-sided CIs and tests ($H_0: \mu=0, H_1: \mu > 0$). And I'll stick to percentile CIs for simplicity (suppose we can assume they will be good in this case):

Suppose the null ($\mu=0$) is true. For a $(1-x)\%$ CI, it will miss zero, $x$% of the time, giving $x\%$ falsely significant results (at the $x\%$ level).

Take a tiny $\epsilon$. If the $(x+\epsilon)$th bootstrap percentile lies just above our null (0), then our null (0) lies outside the CI, it is a false positive and statistically significant at the $(x+\epsilon)\%$ level, and $p<(x+\epsilon)$. Conversely, if the $(x-\epsilon)$th percentile lies just below zero, then $p>(x-\epsilon)$. So surely, for that $x$, $(x-\epsilon)<p=x<(x+\epsilon)$? I.e. $p=x$ is the percentile of the null in the bootstrap samples. Is there a gap in this logic? Or is there another reason why we need to do all this shifting around of data to recreate the null distribution?


Hi: I think what you're missing is that, for bootstrapping to work, not only does the distribution of the "thing" being bootstrapped have to converge to a distibution under the null but that "thing" has to be pivotal. By pivotal, it is meant that the statistic being bootstrapped doesn't depend on the parameter being tested under the null.

But, if we use the bootstrapped samples themselves, then, clearly, that's not true. If we generate samples from the original population, then the bootstrapped distribution of the sample clearly depends on the value of $\mu$.

The idea of bootstrapping is to be able to avoid a distributional assumption about the original sample by using the fact that the constructed pivotal statistic from the sample ( hopefully) converges to a distribution. This way, we can look at the resulting distribution of the pivotal statistic and see where the actual statistic from the original sample falls within that distribution. I hope that helps.

  • $\begingroup$ Thank you for your answer! I've given you an upvote and am not reading up on what it means to be pivotal. $\endgroup$ – justme May 7 '20 at 13:36
  • 1
    $\begingroup$ (That should say NOW reading up...!) $\endgroup$ – justme May 7 '20 at 14:44
  • $\begingroup$ OK, so I'm new to all this, so apologies if this is a stupid question. I think when our "thing" is pivotal, then percentile CIs are good? In that case, is that covered in the OP, by "suppose we can assume percentile CIs wil be good"? If so, can we not cover the more general case by inverting the BCa formulae to convert the percentile of our null in the BS distribution into a "desired alpha" for a CI that would just miss it -- and take that alpha as a p-value? -- possibly there is again something missing in my logic. $\endgroup$ – justme May 7 '20 at 16:42
  • 1
    $\begingroup$ @justme This page describes different types of bootstrapped CI. BCa provides "second-order accurate" CI for non-pivotal statistics. Those CI values, however, are based on the data, without respect to a null hypothesis against which p-values are calculated. It's not that the CI are invalid for BCa, it's the inversion from the CI to a null-hypothesis-based p-value that can't be trusted without a pivotal statistic. See this page about such inversions. And with good CI, why do you need a p-value? $\endgroup$ – EdM May 9 '20 at 19:59
  • 1
    $\begingroup$ @justme I think you understand me well enough, although the distinction is really between a statistic based on sampling from the population you have and a statistic based on sampling from a hypothetical population in which the null hypothesis is true. I assume you certainly can say that the hypothesized null value lies outside the 95% confidence intervals based on the data, which should be good enough. I suppose you could go for more stringent CI (98%) and make a stronger point. $\endgroup$ – EdM May 9 '20 at 22:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.