# Distribution of p-values from multiple experiments

My experimental setup is the following:

1. compute an optimal solution $S$ to problem $A$ on instance $I$,
2. evaluate test statistic $f(S)$,
3. compute $f(S_i)$ for $S_i$ $(0 \leq i \leq n)$ drawn from a uniform distribution of feasible solutions to $A$, and
4. compute a p-value for S.

The null hypothesis is that $f$ is not correlated with the optimality of $A$. The alternative is that $f$ is. The p-value for $S$ is then $l/n$ where $l$ is the number of $S_i$s such that $f(S_i) > f(S)$.

If we repeat this test for many instances $I_j$, we get a distribution of p-values. I would like to determine if we can reject the null hypothesis over all tests.

I've looked at Fisher's method, but I have some pvalues that are so small, that the result is 0.

If I make a histogram of the pvalues I get the following: There is a clear bias towards very small p-values, but what worries me is the uniform looking distribution of the other p-values. Does our data reject the null hypothesis?

EDIT: What if the left-most bar indicated less tests with such small pvalues. How would we evaluate the following distribution of p-values? • The question is "is that distribution skewed toward small values?" (against the null of uniformity). The answer is "it sure is". May 23, 2013 at 8:36

If the null hypothesis is true then the p-values of a test should follow a continuous standard uniform distribution.

Think of it this way: Say we a priori decided a significance level of .05 (5%). That would mean that if we were to repeat our experiment many times and our null hypothesis is true we want to (incorrectly) reject our true null hypothesis in 5% of the replications. So we want to find 5% of the p-values with a value less than .05. Similarly, if we a priori chose our significance level to be .10 we would want to find a p-value less than .10 in 10% of the replications. In principle, there is nothing special about the 5% or 10% significance levels other than convention. So we would want this relationship to hold for any significance level $\alpha$. That way you reject a true null hypothesis $100 \alpha\%$ for all possible values of $\alpha$. This means that we want the following to be true if the null hypothesis is true:

$\mathrm{Pr}(p < \alpha) = \alpha$

This is just the CDF of the continuous standard uniform distribution. A clear deviation from the uniform distribution is an indication that your null hypothesis is to be rejected.

Your graph shows clearly that the distribution of p-values is not uniformly distributed, which would lead me to reject the null hypothesis. The fact that there are some replications where the p-value is very large fits within the logic of statistical testing: a small p-value does not give you absolute certainty that the null hypothesis is false, it just tells you that it is unlikely, but not impossible, that the data you have seen would occure if the null hypothesis were true.

As an asside, you can use this feature to check whether a test performs as it should. For example, many tests are based on an asymptotic argument and you may wonder if your data has enough observations for that argument to kick in. In that case you can simulate data such that you know that the null hypothesis is true but is otherwise similar to your data (same number of observations, independent variables with the same joint distribution, etc), and than look at the distribution of p-values. If you see a deviation from the uniform distribution, then that is an indication that the test is not quite working for your data. Here is an example: https://stats.stackexchange.com/questions/59091#59091

• Understood. Thanks for the nice answer. The histogram I included makes the pattern far too obvious I suppose. I've added another histogram that is less clear. At what point do we have to start doing other tests, and what tests would we do? May 23, 2013 at 19:44
• I would not use an histogram because the binning can add patterns to the graph that are not present in the data. Instead I would plot but the deviation between the observed coverage (the proportion of instances in which the p-value is less than the nominal siginficance) and the nominal significance against the nominal siginficance. You can add a confidence interval to such a graph, like I have done here: <maartenbuis.nl/software/simpplot.html> May 27, 2013 at 7:29