What conditions are there on a (continuous) test statistic for the $p$-value to be uniformly distributed?

Question

I have been trying to prove theorem 10.14 in Wasserman's All of Statisticts, which reads:

If the test statistic has a continuous distribution, then under $ H_0 : \theta = \theta_0 $, the $p$-value has a $ Uniform(0, 1) $ distribution.

Using the definition given for $p$-value and the size $\alpha$ of the test/power $\beta(\theta)$ of a test, this means that if there's a rejection region $ R_\alpha $ for each $ \alpha \in (0, 1) $, and a test statistic $T(X)$, we have:

$$ \text{p-value} = \inf \left\{ \mathbb{P} ( T(X) \in R_\alpha \right\} $$

Now, if we use the "usual" rejection region,

$$ R = \left\{ x : T(X) > c \right\} $$

for a scalar-valued $ T $, meaning the $R_\alpha$s are just open intervals $ (c, \infty) $. Then $ T $ has a CDF $ F $, and we use $ P = F(T) $ and the universality of the uniform.

However, for $ T $s that aren't scalar valued, or $ R_\alpha$s that don't amount to exceeding some critical value, I'm not sure if the result still holds, and if it does still hold, how to prove it.

EDIT: Upon further reflection, I know how to prove it if I can assume that $ R_\alpha \subset R_{\alpha'} $ when $ \alpha < \alpha' $, which seems like a desirable property for a test statistic, but is not obviously a requirement based on $ T $ being continuous (or anything else I can think of like the definition of a probability).

Answer 1

The test statistic has to be continuous and exact. So a t test works fine for normal data matching the null mean:

set.seed(2020)
pv = replicate(10^5, t.test(rnorm(10, 100, 15), mu=100)$p.val)
mean(pv <= .05)
[1] 0.04953

hist(pv, prob=T, col="skyblue2", main="")
 curve(dunif(x), add=T, n=10001, col="orange", lwd=2)

The bar at far left corresponds to the significance level 5%.

However, the Shapiro-Wilk test statistic does not give a precisely uniform plot---even though its rejection rate for normal data is very nearly 5%.

set.seed(2020)
pv = replicate(10^5, shapiro.test(rnorm(10, 100, 15))$p.val)
mean(pv <= .05)
[1] 0.04847
hist(pv, prob=T, col="skyblue2", main="")
   curve(dunif(x), add=T, n=10001, col="orange", lwd=2)

Addendum: The conditions of the test need to be met. For example, in a pooled 2-sample t test, group population variances must be equal (as well as the means, as explicitly specified in the null hypothesis).

set.seed(911)
pv = replicate(10^5, t.test(rnorm(10,0,20),
                            rnorm(20,0,5), var.eq=T)$p.val)
hist(pv, prob=T, col="skyblue2", main="")
 curve(dunif(x), add=T, n=10001, col="orange", lwd=2)

A Welch test is useful even when population variances are grossly unequal; it has very nearly the intended 5% significance level. Also, a histogram of its P-values is hard to distinguish from uniform, but it is not precisely uniform because the Welch t statistic is an approximation.