Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm not a mathematician and I need to prove what this theorem says. I think is easy and I know how it works, but definitively I'm not too much rigorous to make a demonstration. Can anybody help me?

I enounce it:

If the test statistic has a continuous distribution, then under H0 : θ = θ0, the p-value has a Uniform[0, 1] distribution. Therefore, if we reject H0 when the p-value is less than α, the probability of a type I error is α.

  • Therefore, when H0 is true, the p-value is like a random draw from a Uniform[0, 1].

  • On the other hand, if H0 is not true, the distribution of the p-value will tend to concentrate closer to 0.

  • A large p-value can occur for two reasons:

    • H0 is true, or
    • H0 is false, but the test has low power
  • Do not confuse the p-value with P(H0 | data). The p-value is not the probability that the null hypothesis is true.

Thank you so much!

share|improve this question

1 Answer 1

up vote 2 down vote accepted

Your theorem basically follows from the result in theoretical statistics that, for a continuous random variable X with cumulative distribution function F, F(X) has a uniform distribution. That's because your p-value is $1-F_s(S)$, where $S$ is the test statistic and $F_s$ is the distribution of $S$ under the null hypothesis.

The proof goes as follows:

Assume $X$ has cdf $F$. Then

$$ Prob(F(X) \leq t) = Prob(X \leq F^{-1}(t))=F(F^{-1}(t))=t$$

And that's the definition of a uniform: $Prob(U \leq t) = t$, for $t \in [0.1]$.

Why does it matter that $X$ be continuous? If $X$ has discrete point masses of probability, then $F$ may not have an inverse for every possible value of $t$ in the unit interval.

A well-behaved, continuous random variable will have a strictly monotone cdf over its domain, and the inverse will be well defined.

I'm not so sure about the statement that the distribution of the p-value tends towards 0 always and everywhere when the null is false. Consider a contingency table with overly regular cell counts. The chi-square value will be very small and the p-value close to one.

I admit that's an artificial example, and works partly because the chi-squared test is one-sided.

I would qualify your point about the reasons for large p-values by adding "the test was not designed to detect the alternative that actually occurred." In effect, that's the problem with the chi-squared test on a contingency table - it's not designed to detect "rigged" data with overly even cell distributions.

share|improve this answer
    
What means "overly regular cell counts"? Isn't it called unbiased test if the distribution of its p-value tend to 0 whenever the null hypothesis is false? –  Horst Grünbusch Aug 8 at 8:02
    
Overly regular cells would be cells with equal counts or nearly equal counts. This constitutes a departure from multinomial data with equal probabilities, since in practice, one typically gets a lot of variation from cell to cell. A p-value is calculated with reference both to the null and alternate hypotheses. So if the alternate is one-sided, the test is incapable of detecting an alternative from the wrong side. –  Placidia Aug 8 at 16:00

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.