Why does Fisher's method yield $p\gg 0.5$ when combining several p-values all equal to $0.5$?

Question

W = 0
for p in xrange(10):
    W += -2*log(0.5)
print "W:",W
final_p = 1 - scipy.stats.chi2.cdf(W, 2*10)
print "Final p-value",final_p

Why does this result in a final p-value of 0.84, when I used only the most insignificant p-values (p=0.5 for the null hypothesis)? Shouldn't this result in 0.5 as well? Because if you have done tests which say absolutely nothing (=the meaning of p=0.5), don't you want a conclusion which says absolutely nothing as well?

Do I miss something on Fisher's method or p-values?

Answer 1

The way that Fisher's approach measures the combined effect of p-values is to effectively look at their product (the ordering of possible statistics when adding the logs is the same as when taking the product). It then asks whether this is unusually low compared to what you'd find with random p-values when the null is true (which would be draws from a uniform distribution in that case).

In the product, very small values "pull down" the value more than very large values push it up (compared to a typical value). A large probability can't be above 1, but a small one can be very small indeed.

By that product metric, a product of many 0.5's is unusual compared to a product of random uniform values. If your results were really showing nothing, you should really see some small p's in there, but you don't have any. By collecting a lot of 0.5's you're basically getting into the 'even less discrepant than randomness' territory ... which still wouldn't cause you to reject, of course.

The histogram is of the Fisher combined p-value for a sample of 1000 sets of 10 random (uniform) p-values, the green curve is the true density, that for a $\chi^2_{20}$, while the brown line marks the position for the combined $p$ when there are 10 values, each with $p=0.5$.

Note that large values - values in the right tail - are highly significant. The set of ten $0.5$ values is well into the left tail, so they don't indicate significance.

While Fisher's method has much to commend it (not least that it makes a lot of intuitive sense to work with a product of independent p-values), there's nothing actually sacrosanct about that metric. You could, for example, add p-values, and compare that sum to the distribution of a sum of random p-values. By that metric, a lot of p=0.5's would give you a value right in the middle. (There are many other ways one might combine p-values. I mostly just go with Fisher, though, it usually captures what I want a "combined p-value" to capture.)

Answer 2

A p-value of 0.5 is not the "most insignificant p-value". It says that if the null hypothesis is true, than 50% of the time you would expect a result that is at least as extreme as the observed result. You could call that a "typical" result under the null hypothesis, but certainly not "most insignificant". A p-value of 1 would be the "most insignificant p-value".