6
$\begingroup$

I have seen this, and this and a few YouTube videos, and I'm still stuck.

I understand how the probability integral transform gives rise to the result that the CDF of the p-values will have a uniform distribution.

What I don't understand is why that implies that the p-values themselves have a uniform distribution.

That is, I understand this much:

Suppose X ~ Unif(a, b). Then the CDF of X is:

$$P(X \le x) = \begin{cases} 0\ \ {\rm{if}}\ \ x \le a \\ (x-a)/(b-a)\ \ {\rm{if}} \ \ a \le x < b \\ 1\ \ {\rm{if}}\ x \ge b \end{cases}$$

So if X ~ Unif(0, 1), then $$P(X \le x) = x$$ (just substituting a=0 and b=1).

Now suppose $$Y = F(X)$$, and we want to know the probability distribution of Y. That is, we want to know the probability distribution of the CDF of X.

We know that the CDF of a distribution is a unique identifier of a distribution, so if you see, for example, $P(X \le x) = x$ then you know X ~ Unif(0, 1).

We also know that CDFs are right-continuous, and they go from 0 to 1. So it's reasonable to pick a value, f, that lies between 0 and 1 and try to find the probability that the CDF, Y, takes a value less than or equal to f:

$$\begin{align*} P(Y \le f) &= P(F(X) \le f) \\ &= P(X \le F^{-1}(f)) \ {\rm{assuming\ F\ is\ invertible}} \\ &= F(F^{-1}(f)) \\ &= f \end{align*}$$

So since $P(Y \le f) = f, Y = F(X)$ must follow a uniform distribution.

This implies that for any continuous random variable (that satisfies some properties that I'm not sure of), the CDF of that continuous random variable will have a Unif(0, 1) distribution.

It does NOT imply that the random variable itself has a Unif(0, 1) distribution. That is, it does not mean that X has a Unif(0, 1) distribution, only that F(X) has a Unif(0, 1) distribution.

So if a test statistic has a continuous distribution, then the CDF of that test statistic has a Unif(0, 1) distribution. Why does this mean that the p-values have a uniform distribution?

Wait...are p-values the CDF of a test statistic?

Clearly I'm tying myself in knots here. Any help would be appreciated.

EDIT (responding to a comment):

Here's my line of thinking since sleeping on it.

If we have $P(X \le x) = x$, then X ~ Unif(0,1).

Since $P(F(X) \le f) = f$, that means $F(X)$ ~ Unif(0,1), right?

But why does this lead us to think that p-values are uniformly distributed if the null hypothesis is true?

Suppose for instance we have:

$$H_0: \mu \ge 0$$ $$H_a: \mu < 0$$,

and $\sigma$ is known. Let $ts$ be the test statistic, which has a non-standard normal distribution. After standardization, let the z-score associated with the test statistic be $z_{ts}$.

Then we would reject $H_0$ if $P(Z < z_{ts}) < 0.05$. That is, we would reject $H_0$ if the p-value is less than 0.05.

The form $P(Z < z_{ts})$ is the same kind of form as a CDF, right? If the test statistic is continuous then this is the same as $P(Z \le z_{ts})$.

Now let $F(Z) = P(Z \le z_{ts})$.

Is this really a CDF? If so, then what?

What about when we have other alternative hypotheses (like $H_a: \mu > 0$ or $H_a: \mu \ne 0$)?

$\endgroup$
  • $\begingroup$ What exactly do you mean by "the CDF of the p-values will have a uniform distribution"? By definition, the CDF is a function from $\mathbb R$ to $[0,1].$ Are you positing a uniform distribution of such functions? Do you really mean that the CDF is a uniform distribution? If so, your question appears pointless, because it asks whether a uniform distribution is a uniform distribution. $\endgroup$ – whuber Nov 13 '19 at 13:52
  • $\begingroup$ Hello @whuber, please see the edit, I'm trying to explain my question. $\endgroup$ – StatsSorceress Nov 13 '19 at 14:08
  • $\begingroup$ See equation 1 of my answer here stats.stackexchange.com/questions/425425/… $\endgroup$ – ryu576 Nov 14 '19 at 17:10
  • $\begingroup$ Thank you @ryu576, but I'm still confused. The CDF of a continuous test statistic will have a U(0, 1) distribution; but why does that mean p-values are uniformly distributed if H_0 is true? $\endgroup$ – StatsSorceress Nov 17 '19 at 23:10
  • $\begingroup$ @StatsSorceress - added an answer. Let me know if this helps. $\endgroup$ – ryu576 Nov 17 '19 at 23:58
2
$\begingroup$

In hypothesis testing, we compute the test statistic and ask, 'what is the probability of seeing something as or more extreme than this observation'.

Consider a test where the alternate hypothesis is something being 'greater'. In the context of greater alternate, this becomes probabilities of seeing the observed test stat or anything greater than it.

In other words, the p_value is the survival function of the test statistic under the null. So, if our test statistic is $x$ and the null hypothesis involves it distributed according to $X_0$, the p_value becomes (for the test where the alternate is 'greater' and assuming $S_{X_0}$ is the survival function of $X_0$):

$$q=P(\text{Observation as or more extreme than x under null in direction of alternate})$$

$$=P(X_0>x)=S_{X_0}(x)$$

But if the null hypothesis is true, the test statistic, $x$ itself is drawn from the distribution of the null. And we said the distribution of the test statistic under the null is $X_0$. The distribution of the p_value is then given by a random variable $Q$ such that:

$$Q=S_{X_0}(X_0)$$

But we know that if we apply the survival function (or CDF) of a random variable to itself, we get a U(0,1) distribution. This is the basis of the inverse transform sampling technique and Q-Q plots.

Here is a proof:

$$P(Q<q)=P(S_{X_0}(X_0)<q)=P(X_0>S_{X_0}^{-1}(q))=S_{X_0}(S_{X_0}^{-1}(q))=q$$

Where we used in the third expression the fact that the survival function is monotonically decreasing.

But if $P(Q<q)=q$ then $Q$ must be $U(0,1)$.

$\endgroup$
  • $\begingroup$ Thank you very much for this explanation. I think I follow most of it, But I'm stuck here: saying that t comes from the distribution X_0 is not the same as saying t=X_0, I think. So going from equation 1 to equation 2 I'm still lost. $\endgroup$ – StatsSorceress Nov 18 '19 at 1:39
  • $\begingroup$ Right, the p_value in the first equation is a deterministic number. The one in the second equation is a random variable. Perhaps the second one should be capitalized then. $\endgroup$ – ryu576 Nov 18 '19 at 1:42
  • $\begingroup$ OH! That's brilliant! :D So since a p-value is a CDF (well, a survival function in this case, but close enough) of a continuous random variable, then the p-values must have a uniform distribution. That gives rise to more questions, but that definitely solves the question I asked. Thank you! $\endgroup$ – StatsSorceress Nov 18 '19 at 1:49
  • $\begingroup$ Yeah, not the survival function of just any continuous random variable. The survival function of the same random variable that is going to be passed to it as argument (which is $X_0$). This holds for discrete as well as continuous random variables. Also, it is uniform only under the null. Under the alternate (with some assumption on effect size), we have $P=S_{X_0}(X_a)$ which is not uniform. $\endgroup$ – ryu576 Nov 18 '19 at 1:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.