# Why does Fisher's method yield $p\gg 0.5$ when combining several p-values all equal to $0.5$?

W = 0
for p in xrange(10):
W += -2*log(0.5)
print "W:",W
final_p = 1 - scipy.stats.chi2.cdf(W, 2*10)
print "Final p-value",final_p


Why does this result in a final p-value of 0.84, when I used only the most insignificant p-values (p=0.5 for the null hypothesis)? Shouldn't this result in 0.5 as well? Because if you have done tests which say absolutely nothing (=the meaning of p=0.5), don't you want a conclusion which says absolutely nothing as well?

Do I miss something on Fisher's method or p-values?

• Note that Fisher's exact test has nothing to do with Fisher's method for combining p-values, so I removed your tag May 6, 2014 at 19:30
• I've taken the liberty of adding the python tag. Where possible one should indicate the language you give code in, both for those people that don't instantly recognize it, and also to facilitate searches. May 6, 2014 at 22:50
• "Because if you have done tests which say absolutely nothing (=the meaning of p=0.5)"-- this is not the correct interpretation of $p=0.5$, especially not for two tailed tests. Tests which are completely consistent with the null hypothesis have $p=1$. Aug 12, 2015 at 22:16

The way that Fisher's approach measures the combined effect of p-values is to effectively look at their product (the ordering of possible statistics when adding the logs is the same as when taking the product). It then asks whether this is unusually low compared to what you'd find with random p-values when the null is true (which would be draws from a uniform distribution in that case).

In the product, very small values "pull down" the value more than very large values push it up (compared to a typical value). A large probability can't be above 1, but a small one can be very small indeed.

By that product metric, a product of many 0.5's is unusual compared to a product of random uniform values. If your results were really showing nothing, you should really see some small p's in there, but you don't have any. By collecting a lot of 0.5's you're basically getting into the 'even less discrepant than randomness' territory ... which still wouldn't cause you to reject, of course.

The histogram is of the Fisher combined p-value for a sample of 1000 sets of 10 random (uniform) p-values, the green curve is the true density, that for a $\chi^2_{20}$, while the brown line marks the position for the combined $p$ when there are 10 values, each with $p=0.5$.

Note that large values - values in the right tail - are highly significant. The set of ten $0.5$ values is well into the left tail, so they don't indicate significance.

While Fisher's method has much to commend it (not least that it makes a lot of intuitive sense to work with a product of independent p-values), there's nothing actually sacrosanct about that metric. You could, for example, add p-values, and compare that sum to the distribution of a sum of random p-values. By that metric, a lot of p=0.5's would give you a value right in the middle. (There are many other ways one might combine p-values. I mostly just go with Fisher, though, it usually captures what I want a "combined p-value" to capture.)

A p-value of 0.5 is not the "most insignificant p-value". It says that if the null hypothesis is true, than 50% of the time you would expect a result that is at least as extreme as the observed result. You could call that a "typical" result under the null hypothesis, but certainly not "most insignificant". A p-value of 1 would be the "most insignificant p-value".

• Combining P-values usually involves the use of one-tailed P-values and so p=0.5 is what you get when the data maximally agree with the null hypothesis. ("Most insignificant" is an awful description.) May 6, 2014 at 22:01
• @MichaelLew are you thinking of two tailed tests? Suppose you are interested in the test of $\mathcal{H}_0: \mu \le 0$ and you simulate 100 realizations of $X \sim \mathcal{N}(-3, 1)$ I type t.test(x = rnorm(100, -3, 1), alternative = 'greater', mu = 0) and obtain $p \approx 1$ which makes sense intuitively. Aug 12, 2015 at 22:17