Bootstrap p-value

Question

Assume you created $B$ bootstrap replicates $T^{(i)}, i\in\{1,\dots,B\}$ for a test statistic $T$ since you don't know the distribution of this test statistic.

Why is the p-value approximated by $\frac 1 B \sum_{i=1}^B\mathbb 1_{\{T^{(i)}> T\}}$? Why not for example by $\frac 1 B \sum_{i=1}^B \mathbb 1_{\{T^{(i)}<T\}}$? Can anyone explain the intuition behind this?

Answer 1

Actually, both are possible. The definition from Wikipedia in the other answer is somewhat imprecise.

Here, it is necessary to point out that "greater magnitude" in the definition

The probability for a given statistical model that, when the null hypothesis is true, the statistical summary would be the same as or of greater magnitude than the actual observed results.

needs to be taken to mean "more extreme" in the sense of as or more unlikely when the null is true.

Hence, when you conduct a right-tailed test, large values of $T$ provide evidence against (are unlikely if) the null (is true). Conversely, small (i.e., large negative) values are by no means surprising.

This, if bootstrap test statistics $T_i$ can be seen as draws from the null distribution (i.e., your bootstrap was successful), $\frac 1 B \sum_{i=1}^B\mathbb 1_{\{T^{(i)}> T\}}$ would be a useful bootstrap p-value.

Conversely, if you test against left-tailed alternatives, $\frac 1 B \sum_{i=1}^B \mathbb 1_{\{T^{(i)}<T\}}$ would be appropriate, as it would give you the fraction of draws from your approximation to the null distribution that are more extreme than (less compatible with) the null than the observed test statistic $T$.

Answer 2

Go back to the definition of the P-value (from Wikipedia ):

The probability for a given statistical model that, when the null hypothesis is true, the statistical summary would be the same as or of greater magnitude than the actual observed results.

For this reason, its estimation by bootstrap is the proportion of the $T^{(i)}$ that exceed $T$.