6
$\begingroup$
W = 0
for p in xrange(10):
    W += -2*log(0.5)
print "W:",W
final_p = 1 - scipy.stats.chi2.cdf(W, 2*10)
print "Final p-value",final_p

Why does this result in a final p-value of 0.84, when I used only the most insignificant p-values (p=0.5 for the null hypothesis)? Shouldn't this result in 0.5 as well? Because if you have done tests which say absolutely nothing (=the meaning of p=0.5), don't you want a conclusion which says absolutely nothing as well?

Do I miss something on Fisher's method or p-values?

$\endgroup$
3
  • 2
    $\begingroup$ Note that Fisher's exact test has nothing to do with Fisher's method for combining p-values, so I removed your tag $\endgroup$
    – Aniko
    May 6 '14 at 19:30
  • 3
    $\begingroup$ I've taken the liberty of adding the python tag. Where possible one should indicate the language you give code in, both for those people that don't instantly recognize it, and also to facilitate searches. $\endgroup$
    – Glen_b
    May 6 '14 at 22:50
  • $\begingroup$ "Because if you have done tests which say absolutely nothing (=the meaning of p=0.5)"-- this is not the correct interpretation of $p=0.5$, especially not for two tailed tests. Tests which are completely consistent with the null hypothesis have $p=1$. $\endgroup$
    – AdamO
    Aug 12 '15 at 22:16
13
$\begingroup$

The way that Fisher's approach measures the combined effect of p-values is to effectively look at their product (the ordering of possible statistics when adding the logs is the same as when taking the product). It then asks whether this is unusually low compared to what you'd find with random p-values when the null is true (which would be draws from a uniform distribution in that case).

In the product, very small values "pull down" the value more than very large values push it up (compared to a typical value). A large probability can't be above 1, but a small one can be very small indeed.

By that product metric, a product of many 0.5's is unusual compared to a product of random uniform values. If your results were really showing nothing, you should really see some small p's in there, but you don't have any. By collecting a lot of 0.5's you're basically getting into the 'even less discrepant than randomness' territory ... which still wouldn't cause you to reject, of course.

enter image description here

The histogram is of the Fisher combined p-value for a sample of 1000 sets of 10 random (uniform) p-values, the green curve is the true density, that for a $\chi^2_{20}$, while the brown line marks the position for the combined $p$ when there are 10 values, each with $p=0.5$.

Note that large values - values in the right tail - are highly significant. The set of ten $0.5$ values is well into the left tail, so they don't indicate significance.

While Fisher's method has much to commend it (not least that it makes a lot of intuitive sense to work with a product of independent p-values), there's nothing actually sacrosanct about that metric. You could, for example, add p-values, and compare that sum to the distribution of a sum of random p-values. By that metric, a lot of p=0.5's would give you a value right in the middle. (There are many other ways one might combine p-values. I mostly just go with Fisher, though, it usually captures what I want a "combined p-value" to capture.)

$\endgroup$
3
$\begingroup$

A p-value of 0.5 is not the "most insignificant p-value". It says that if the null hypothesis is true, than 50% of the time you would expect a result that is at least as extreme as the observed result. You could call that a "typical" result under the null hypothesis, but certainly not "most insignificant". A p-value of 1 would be the "most insignificant p-value".

$\endgroup$
2
  • 2
    $\begingroup$ Combining P-values usually involves the use of one-tailed P-values and so p=0.5 is what you get when the data maximally agree with the null hypothesis. ("Most insignificant" is an awful description.) $\endgroup$ May 6 '14 at 22:01
  • $\begingroup$ @MichaelLew are you thinking of two tailed tests? Suppose you are interested in the test of $\mathcal{H}_0: \mu \le 0$ and you simulate 100 realizations of $X \sim \mathcal{N}(-3, 1)$ I type t.test(x = rnorm(100, -3, 1), alternative = 'greater', mu = 0) and obtain $p \approx 1$ which makes sense intuitively. $\endgroup$
    – AdamO
    Aug 12 '15 at 22:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.