Recently, I have found in a paper by Klammer, et al. a statement that p-values should be uniformly distributed. I believe the authors, but cannot understand why it is so.

Klammer, A. A., Park, C. Y., and Stafford Noble, W. (2009) Statistical Calibration of the SEQUEST XCorr Function. Journal of Proteome Research. 8(4): 2106–2113.

  • 24
    $\begingroup$ This is immediate from the definition of the p-value as the probability integral transform of the test statistic using the distribution under the null hypothesis. The conclusion requires that the distribution be continuous. When the distribution is discrete (or has atoms), the distribution of p-values is discrete, too, and therefore can only approximately be uniform. $\endgroup$ – whuber May 10 '11 at 18:46
  • 2
    $\begingroup$ @whuber gave the answer which was something I suspected. I asked the original reference just to be sure that something was not lost in translation. Usually it does not matter whether the article is specific or not, statistical content always shows through :) $\endgroup$ – mpiktas May 10 '11 at 18:56
  • 13
    $\begingroup$ Only when $H_0$ is true! ... and more strictly, only when continuous (though something like it is true in the non-continuous case; I don't know the right word for the most general case; it's not uniformity). Then it follows from the definition of p-value. $\endgroup$ – Glen_b Jun 7 '13 at 1:35
  • 4
    $\begingroup$ This could be seen as a variant of the fundamental statistical mechanics principle (that students often have similar difficulty accepting) that all micro-states of a physical system have equal probability. $\endgroup$ – DWin Jul 21 '13 at 19:43
  • 6
    $\begingroup$ How about the claim in this article: plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0076010 ? $\endgroup$ – user54876 Aug 28 '14 at 18:26

To clarify a bit. The p-value is uniformly distributed when the null hypothesis is true and all other assumptions are met. The reason for this is really the definition of alpha as the probability of a type I error. We want the probability of rejecting a true null hypothesis to be alpha, we reject when the observed $\text{p-value} < \alpha$, the only way this happens for any value of alpha is when the p-value comes from a uniform distribution. The whole point of using the correct distribution (normal, t, f, chisq, etc.) is to transform from the test statistic to a uniform p-value. If the null hypothesis is false then the distribution of the p-value will (hopefully) be more weighted towards 0.

The Pvalue.norm.sim and Pvalue.binom.sim functions in the TeachingDemos package for R will simulate several data sets, compute the p-values and plot them to demonstrate this idea.

Also see:

Murdoch, D, Tsai, Y, and Adcock, J (2008). P-Values are Random Variables. The American Statistician, 62, 242-245.

for some more details.


Since people are still reading this answer and commenting, I thought that I would address @whuber's comment.

It is true that when using a composite null hypothesis like $\mu_1 \leq \mu_2$ that the p-values will only be uniformly distributed when the 2 means are exactly equal and will not be a uniform if $\mu_1$ is any value that is less than $\mu_2$. This can easily be seen using the Pvalue.norm.sim function and setting it to do a one sided test and simulating with the simulation and hypothesized means different (but in the direction to make the null true).

As far as statistical theory goes, this does not matter. Consider if I claimed that I am taller than every member of your family, one way to test this claim would be to compare my height to the height of each member of your family one at a time. Another option would be to find the member of your family that is the tallest and compare their height with mine. If I am taller than that one person then I am taller than the rest as well and my claim is true, if I am not taller than that one person then my claim is false. Testing a composite null can be seen as a similar process, rather than testing all the possible combinations where $\mu_1 \leq \mu_2$ we can test just the equality part because if we can reject that $\mu_1 = \mu_2$ in favour of $\mu_1 > \mu_2$ then we know that we can also reject all the possibilities of $\mu_1 < \mu_2$. If we look at the distribution of p-values for cases where $\mu_1 < \mu_2$ then the distribution will not be perfectly uniform but will have more values closer to 1 than to 0 meaning that the probability of a type I error will be less than the selected $\alpha$ value making it a conservative test. The uniform becomes the limiting distribution as $\mu_1$ gets closer to $\mu_2$ (the people who are more current on the stat-theory terms could probably state this better in terms of distributional supremum or something like that). So by constructing our test assuming the equal part of the null even when the null is composite, then we are designing our test to have a probability of a type I error that is at most $\alpha$ for any conditions where the null is true.

| cite | improve this answer | |
  • 2
    $\begingroup$ The article "P-Values are Random Variables" is really interesting, is there any introductory book that adheres to the principles stated in the article? $\endgroup$ – Alessandro Jacopson Jun 30 '11 at 13:09
  • $\begingroup$ @uvts_cvs, I think most intro books follow the general idea, but I don't know of any that make it as explicit as the article. The theory books are more likely to talk about how the p-value is a transform from the statistic to something that is uniform under the null. $\endgroup$ – Greg Snow Jun 30 '11 at 15:34
  • 9
    $\begingroup$ Despite the comment I posted to the question, I have since realized that the conclusion is not true except in special cases. The problem occurs with composite hypotheses, such as $\mu_1 \le \mu_2$. "The null hypothesis is true" now covers many possibilities, such as the case $\mu_1 = \mu_2 - 10^6$. In such a case, the p-values will not be uniformly distributed. I suspect one could manufacture (somewhat artificial) situations in which, no matter what element of the null hypothesis holds, the distribution of p-values would never be anywhere near uniform. $\endgroup$ – whuber Jul 20 '12 at 14:50
  • 2
    $\begingroup$ @Greg Snow: I think that the distribution of the p-values is not always uniform, it is uniform when they are computed from a continuous distribution, but not when they are computed from a discrete distribution $\endgroup$ – user83346 Aug 16 '15 at 16:58
  • 2
    $\begingroup$ I have expanded the answer above to address the comment by @whuber. $\endgroup$ – Greg Snow Aug 17 '15 at 15:42

Under the null hypothesis, your test statistic $T$ has the distribution $F(t)$ (e.g., standard normal). We show that the p-value $P=F(T)$ has a probability distribution $$\begin{equation*} \Pr(P < p) = \Pr(F^{-1}(P) < F^{-1}(p)) = \Pr(T < t) \equiv p; \end{equation*}$$ in other words, $P$ is distributed uniformly. This holds so long as $F(\cdot)$ is invertible, a necessary condition of which is that $T$ is not a discrete random variable.

This result is general: the distribution of an invertible CDF of a random variable is uniform on $[0,1]$.

| cite | improve this answer | |
  • 8
    $\begingroup$ you might want to rephrase your last comment, which is a little confusing. Continuous CDFs do not necessarily have a (proper) inverse. (Can you think of a counterexample?) So your proof requires additional conditions to hold. The standard way to get around this is to define the pseudoinverse $F^{\,\leftarrow}(y) = \inf\{x: F(x) \geq y\}$. The argument becomes more subtle, too. $\endgroup$ – cardinal May 26 '11 at 23:36
  • 1
    $\begingroup$ Concerning working with generalized inverses, see link.springer.com/article/10.1007%2Fs00186-013-0436-7 (in particular, F(T) is only uniform if F is continuous -- doesn't matter whether F is invertible or not). Concerning your definition of a p-value: I don't think it's always 'F(T)'. It's the probability (under the null) of taking on a value more extreme than the observed one, so it could also be the survival function (just to be precise here). $\endgroup$ – Marius Hofert Mar 5 '16 at 9:03
  • $\begingroup$ Isn't $F(t)$ the CDF? $\endgroup$ – zyxue May 2 '18 at 21:58
  • 1
    $\begingroup$ @zyxue Yes, the cdf is sometimes referred to as the "distribution". $\endgroup$ – mai Sep 22 '18 at 2:50

Let $T$ denote the random variable with cumulative distribution function $F(t) \equiv \Pr(T<t)$ for all $t$. Assuming that $F$ is invertible we can derive distribution of the random p-value $P = F(T)$ as follows:

$$ \Pr(P<p) = \Pr(F(T) < p) = \Pr(T < F^{-1}(p)) = F(F^{-1}(p)) = p, $$

from which we can conclude that the distribution of $P$ is uniform on $[0,1]$.

This answer is similar to Charlie's, but avoids having to define $t = F^{-1}(p)$.

| cite | improve this answer | |
  • $\begingroup$ As you've defined F, isn't P = F(T) = Pr(T < T) = 0? $\endgroup$ – TrynnaDoStat Jun 27 '19 at 19:24
  • 1
    $\begingroup$ Not exactly, the "syntactic replacement" of $F(T) = \Pr(T<T)$ is somewhat misleading. Formally speaking, $F(T)$ is the random variable defined by $(F(T))(\omega) = F(T(\omega)) := \Pr(T < T(\omega))$ $\endgroup$ – jII Jun 27 '19 at 21:17

Simple simulation of distribution of p-values in case of linear regression between two independent variables :

# estimated model is: y = a0 + a1*x + e

obs<-100                # obs in each single regression
Nloops<-1000            # number of experiments
output<-numeric(Nloops) # vector holding p-values of estimated a1 parameter from Nloops experiments

for(i in seq_along(output)){


# x and y are independent, so null hypothesis is true
output[i] <-(summary(lm(y~x)) $ coefficients)[2,4] # we grab p-value of a1

if(i%%100==0){cat(i,"from",Nloops,date(),"\n")} # after each 100 iteration info is printed


plot(hist(output), main="Histogram of a1 p-values")
ks.test(output,"punif") # Null hypothesis is that output distr. is uniform
| cite | improve this answer | |
  • 7
    $\begingroup$ Could you elaborate on how this answers the question? Although its output illustrates a special case of the assertion, no amount of code would be capable of addressing the question of why? That requires additional explanation. $\endgroup$ – whuber Jun 2 '15 at 14:11

I think the answer as to "Why are p-values uniformly distributed under the null hypothesis?" has been sufficiently discussed from a mathematical perspective. What I thought is missing is a visual explanation of this and the idea of thinking of p-values as areas to the left of a set of quantiles under a given distribution. By quantiles I mean cut-off points along a distribution (in this example the standard normal distribution), which split the distribution into equal parts containing exactly the same area under the curve.

For this example, I generated 100 random data points from the standard normal distribution with a mean of 0 and a standard deviation of 1, $\mathcal{N}(\mu = 0, \sigma = 1)$. Then I plotted those points in a histogram and we can see a bell-shaped distribution forming (Fig. 1A). Then I calculated the p-values of those points, i.e. the areas to the left of those points along the standard normal distribution, plotted those p-values in a histogram (Fig. 1B) and a uniform distribution is emerging binning those p-values in 0.1 intervals.

This step, i.e. the step from Fig 1A to Fig 1B is puzzling for many people and has been for me as well for some time - until I started thinking of p-values as areas under the curve. My thought was that if I split the standard normal distribution into equal chunks containing the same area (in this case 0.1 to match the histogram in Fig 1B), I will have larger intervals in the tails (Fig 1C). Now if I go back to Fig 1A, I can see that I can fit all points ranging from -4 to -1.28 into the first bin of Fig 1B since they all result into areas (or p-values) of less than or equal to 0.1. As the density of points is increasing towards the mean, the intervals that cover an area of 0.1 are becoming increasingly smaller (Fig 1C) but the number of points in those intervals remains roughly equal and in this case matches the count in Fig 1B.

enter image description here

Once I understood this it was also easy for me to explain why a random sample of 100 points from a normal distribution with mean of 0 and a standard deviation of 3, $\mathcal{N}(\mu = 0, \sigma = 3)$ results into a higher frequency of p-values around 0 and 1 or in the tails (Fig 2B). The reason is that the p-values are calculated based on the standard normal distribution yet the sample comes from a normal distribution with mean of 0 and a standard deviation of 3. This will result into many more points in the tails than it would be for a sample coming from the standard normal distribution.

enter image description here

I hope this was not overly confusing and added some value to this thread.

| cite | improve this answer | |

I don't think most of these answers actually answer the question in generality. They are restricted to the case when there is a simple null hypothesis and when the test statistic has an invertible CDF (as in a continuous random variable which has a strictly increasing CDF). These cases are the cases which most people tend to care about with the z-test and t-test, though for testing a binomial mean (for example) one does not have such a CDF. What is provided above seems correct to my eyes for these restricted cases.

If null hypotheses are composite then things are a bit more complicated. The most general proof of this fact I've seen under the composite case using some assumptions regarding rejection regions is provided in Lehmann and Romano's "Testing Statisitical Hypotheses," pages 63-64. I'll try to reproduce the argument below...

We test a null hypothesis $H_0$ versus an alternative hypothesis $H_1$ based upon a test statistic, which we'll denote as the random variable $X$. The test statistic is assumed to come from some parametric class, i.e., $X \sim P_\theta$, where $P_\theta$ is an element of the family of probability distributions $\mathcal{P} \equiv \{P_\theta \mid \theta \in \Theta \}$, and $\Theta$ is a parameter space. The null hypothesis $H_0: \theta \in \Theta_0$ and the alternative hypothesis $H_1: \theta \in \Theta_1$ form a partition of $\Theta$ in that $$ \Theta = \Theta_0 \cup \Theta_1 $$ where $$ \Theta_0 \cap \Theta_1 = \emptyset. $$

The result of the test may be denoted $$ \phi_\alpha(X) = 1_{R_\alpha}(X) $$ where for any set $S$ we define $$ 1_{S}(X) = \begin{cases} 1, & X \in S, \\ 0, & X \notin S. \end{cases} $$ Here $\alpha$ is our significance level, and $R_\alpha$ denotes the rejection region of the test for significance level $\alpha$.

Suppose the rejection regions satisfy the $$ R_\alpha \subset R_{\alpha'} $$ if $\alpha < \alpha'$. In this case of nested rejection regions, it is useful to determine not only whether or not the null hypothesis is rejected at a given significance level $\alpha$, but also to determine the smallest significance level for which the null hypothesis would be rejected. This level is known as the p-value, $$ \hat{p} = \hat{p}(X) \equiv \inf\{\alpha \mid X \in R_\alpha\}, $$ This number gives us an idea of how strong the data (as portrayed by the test statistic $X$) contradict the null hypothesis $H_0$.

Suppose that $X \sim P_\theta$ for some $\theta \in \Theta$ and that $H_0: \theta \in \Theta_0$. Suppose additionally that the rejection regions $R_\alpha$ obey the nesting property stated above. Then the following holds:

  1. If $\sup_{\theta \in \Theta_0} P_\theta(X \in R_\alpha) \leq \alpha$ for all $0 < \alpha < 1$, then for $\theta \in \Theta_0$, $$ P_\theta(\hat{p} \leq u) \leq u \quad \text{for all} \quad 0 \leq u \leq 1. $$

  2. If for $\theta \in \Theta_0$ we have $P_\theta(X \in R_\alpha) = \alpha$ for all $0 < \alpha < 1$, then for $\theta \in \Theta_0$ we have $$ P_\theta(\hat{p} \leq u) = u \quad \text{for all} \quad 0 \leq u \leq 1. $$

Note this first property just tells us that the false positive rate is controlled at $u$ by rejecting when the p-value is less than $u$, and the second property tells us (given an additional assumption) that p-values are uniformly distributed under the null hypothesis.

The proof is as follows:

  1. Let $\theta \in \Theta_0$, and assume $\sup_{\theta \in \Theta_0} P_\theta(X \in R_\alpha) \leq \alpha$ for all $0 < \alpha < 1$. Then by definition of $\hat{p}$, we have $\{\hat{p} \leq u\} \subset \{X \in R_v\}$ for all $u < v$. By monotonicity and the assumption, it follows that $P_\theta(\hat{p} \leq u) \leq P_\theta(X \in R_v) \leq v$ for all $u < v$. Letting $v \searrow u$, it follows that $P_\theta(\hat{p} \leq u) \leq u$.

  2. Let $\theta \in \Theta_0$, and assume that $P_\theta(X \in R_\alpha) = \alpha$ for all $0 < \alpha < 1$. Then $\{X \in R_u\} \subset \{\hat{p}(X) \leq u\}$, and by monotonicity it follows that $u = P_\theta(X \in R_u) \leq P_\theta(\hat{p} \leq u)$. Considering (1), it follows that $P_\theta(\hat{p}(X) \leq u) = u$.

Note that the assumption in (2) does not hold when a test statistic is discrete even if the null hypothesis is simple rather than composite. Take for instance $X \sim \mathrm{Binom}(10, \theta)$ with $H_0: \theta = .5$ and $H_1: \theta > 0.5$. I.e., flip a coin ten times and test whether it's fair vs biased towards heads (encoded as a 1). The probability of seeing 10 heads in 10 fair coin flips is (1/2)^10 = 1/1024. The probability of seeing 9 or 10 heads in 10 fair coin flips is 11/1024. For any $\alpha$ strictly between 1/1024 and 11/1024, you'd reject the null if $X = 10$, but we don't have that $\Pr(X \in R_\alpha) = \alpha$ for those values of $\alpha$ when $\theta = 0.5$. Instead $\Pr(X \in R_\alpha) = 1/1024$ for such $\alpha$.

| cite | improve this answer | |
  • 1
    $\begingroup$ Should clarify that the generality provided in Lehmann and Romano is for general rejection regions. Still you only have "valid" p-values for composite nulls and non-continuous test stats. $\endgroup$ – Adam Apr 30 '19 at 22:44

If p values are uniformly distributed under the H0 that means that it is as likely to see a p-value of .05 as a p-value of .80, but this is not true, as it is less likely to observe a p-value of .05 than a p-value of .80, because that is precisely the definition of the normal distribution from which the p-value are taken. There will be more samples fallling within the range of normality than outside it, by definition. Therefore, more likely to find larger p-values than smaller ones.

| cite | improve this answer | |
  • 3
    $\begingroup$ -1. This is completely wrong. I wonder who upvoted this. P-values under point H0 are uniformly distributed. $\endgroup$ – amoeba Nov 19 '17 at 9:52
  • 2
    $\begingroup$ -1. This doesn't even make enough sense to be called wrong: "range of normality" is meaningless and p-values inherently have nothing to do with normal distributions in the first place. $\endgroup$ – whuber Nov 19 '17 at 16:45

Not the answer you're looking for? Browse other questions tagged or ask your own question.