# What conditions are there on a (continuous) test statistic for the $p$-value to be uniformly distributed?

I have been trying to prove theorem 10.14 in Wasserman's All of Statisticts, which reads:

If the test statistic has a continuous distribution, then under $$H_0 : \theta = \theta_0$$, the $$p$$-value has a $$Uniform(0, 1)$$ distribution.

Using the definition given for $$p$$-value and the size $$\alpha$$ of the test/power $$\beta(\theta)$$ of a test, this means that if there's a rejection region $$R_\alpha$$ for each $$\alpha \in (0, 1)$$, and a test statistic $$T(X)$$, we have:

$$\text{p-value} = \inf \left\{ \mathbb{P} ( T(X) \in R_\alpha \right\}$$

Now, if we use the "usual" rejection region,

$$R = \left\{ x : T(X) > c \right\}$$

for a scalar-valued $$T$$, meaning the $$R_\alpha$$s are just open intervals $$(c, \infty)$$. Then $$T$$ has a CDF $$F$$, and we use $$P = F(T)$$ and the universality of the uniform.

However, for $$T$$s that aren't scalar valued, or $$R_\alpha$$s that don't amount to exceeding some critical value, I'm not sure if the result still holds, and if it does still hold, how to prove it.

EDIT: Upon further reflection, I know how to prove it if I can assume that $$R_\alpha \subset R_{\alpha'}$$ when $$\alpha < \alpha'$$, which seems like a desirable property for a test statistic, but is not obviously a requirement based on $$T$$ being continuous (or anything else I can think of like the definition of a probability).

• In order to prove this you need to assume the distribution of the test statistic is the same for every element of the null hypothesis. When that's the case, this result is essentially a tautology, because the p-value is constructed to have a uniform distribution.
– whuber
Sep 11, 2020 at 22:58
• @whuber I almost understand what you mean about being the same, but not quite. Could you please elaborate on that?
– Dave
Sep 11, 2020 at 23:00

The test statistic has to be continuous and exact. So a t test works fine for normal data matching the null mean:

set.seed(2020)
pv = replicate(10^5, t.test(rnorm(10, 100, 15), mu=100)$p.val) mean(pv <= .05)  0.04953 hist(pv, prob=T, col="skyblue2", main="") curve(dunif(x), add=T, n=10001, col="orange", lwd=2) The bar at far left corresponds to the significance level 5%. However, the Shapiro-Wilk test statistic does not give a precisely uniform plot---even though its rejection rate for normal data is very nearly 5%. set.seed(2020) pv = replicate(10^5, shapiro.test(rnorm(10, 100, 15))$p.val)
mean(pv <= .05)
 0.04847
hist(pv, prob=T, col="skyblue2", main="") Addendum: The conditions of the test need to be met. For example, in a pooled 2-sample t test, group population variances must be equal (as well as the means, as explicitly specified in the null hypothesis).

set.seed(911)
pv = replicate(10^5, t.test(rnorm(10,0,20),
rnorm(20,0,5), var.eq=T)\$p.val)
hist(pv, prob=T, col="skyblue2", main="") 