Prove this theorem related with specification tests

Question

I'm not a mathematician and I need to prove what this theorem says. I think is easy and I know how it works, but definitively I'm not too much rigorous to make a demonstration. Can anybody help me?

I enounce it:

If the test statistic has a continuous distribution, then under H0 : θ = θ0, the p-value has a Uniform[0, 1] distribution. Therefore, if we reject H0 when the p-value is less than α, the probability of a type I error is α.

• Therefore, when H0 is true, the p-value is like a random draw from a Uniform[0, 1].

• On the other hand, if H0 is not true, the distribution of the p-value will tend to concentrate closer to 0.

• A large p-value can occur for two reasons:

  • H0 is true, or
  • H0 is false, but the test has low power

• Do not confuse the p-value with P(H0 | data). The p-value is not the probability that the null hypothesis is true.

Thank you so much!

Answer 1

Your theorem basically follows from the result in theoretical statistics that, for a continuous random variable X with cumulative distribution function F, F(X) has a uniform distribution. That's because your p-value is $1-F_s(S)$, where $S$ is the test statistic and $F_s$ is the distribution of $S$ under the null hypothesis.

The proof goes as follows:

Assume $X$ has cdf $F$. Then

$$ Prob(F(X) \leq t) = Prob(X \leq F^{-1}(t))=F(F^{-1}(t))=t$$

And that's the definition of a uniform: $Prob(U \leq t) = t$, for $t \in [0.1]$.

Why does it matter that $X$ be continuous? If $X$ has discrete point masses of probability, then $F$ may not have an inverse for every possible value of $t$ in the unit interval.

A well-behaved, continuous random variable will have a strictly monotone cdf over its domain, and the inverse will be well defined.

I'm not so sure about the statement that the distribution of the p-value tends towards 0 always and everywhere when the null is false. Consider a contingency table with overly regular cell counts. The chi-square value will be very small and the p-value close to one.

I admit that's an artificial example, and works partly because the chi-squared test is one-sided.

I would qualify your point about the reasons for large p-values by adding "the test was not designed to detect the alternative that actually occurred." In effect, that's the problem with the chi-squared test on a contingency table - it's not designed to detect "rigged" data with overly even cell distributions.