# Understanding Fisher's combined test

I am using Fisher's combined test to fuse several different independent tests. I have a problem understanding the results in some cases.

Example: Let's say I run two different tests, both with the hypothesis that mu is smaller than 0. Let's say that n is identical and the two samples have the same calculated variance. However, let's assume that one test yielded an average that is $1.5$ and the other is $-1.5$. I will get two complementing p-vals (e.g., $0.995$ & $0.005$). Interestingly, combining the two brings about a significant $p$-value in the Fisher test: $p=0.0175$.

This is weird because I could have chosen the exact opposite test $(\mu>0)$ and sampled results - and still get $p=0.0175$. It's almost as if the Fisher test does not take the direction of the hypothesis into account.

Can anyone explain this?

Thanks

• If I interpret this question correctly, the discussion in Rice, A Consensus Combined P-Value Test and the Family-wide Significance of Component Tests (Biometrics 1990) explains this problem: see p. 304. The paper offers a solution.
– whuber
Commented Jun 25, 2012 at 13:08
• Actually using Fisher's combined probability test the combined p for 0.995 and 0.005 is 0.03. Not that it changes the interpretation (smile) but I am wondering where the 0.0175 came from. Commented Jan 19, 2016 at 23:35
• @AussieAndy Yes, I agree -- I make it about 0.03136 Commented Jan 20, 2016 at 1:10

The Fisher combination test is intended to combine information from separate tests done on independent data sets in order to obtain power when the individual tests may not have sufficient power. The idea is that if the $$k$$ null hypotheses are all correct the $$p$$-value will be uniformly distributed on $$[0,1]$$ independently of each other. This means that $$-2 ∑ \log(p_i)$$ will be $$\chi^2$$ with $$2k$$ degrees of freedom. Rejecting this combined null hypothesis leads to the conclusion that at least one of the null hypotheses is false. That is what you are doing when you apply this procedure.

• This does not seem to address the real issue raised by the question: because the two p-values are symmetrically opposite, and therefore (at least according to some intuition) ought to "cancel," how is it that Fisher's method produces a "significant" result--and which conclusion does it support??
– whuber
Commented Jan 19, 2016 at 23:42
• +1 for Rejecting this combined null hypothesis leads to the conclusion that at least one of the null hypotheses is false. Commented Dec 8, 2019 at 19:22
• I think the OP & at the time @whuber in his comment are misunderstanding the meaning of the rejection of the combined null hypotheses. eric_kernfield is emphasizing this by repeating what I said in my answer. Commented Dec 8, 2019 at 19:28
• @Michael, I doubt I misapprehended something as elementary as what it means to reject the combined hypotheses. What is missing from your answer is an explanation of the apparent paradox raised by the OP and in my comment. One place we might seek an explanation is to note that in one case the data were consistent with the null and in the other case they were noticeably inconsistent. The combined dataset thereby still exhibits some inconsistency with the null, which may be why the Fisher p-value is low--but not that low. This deserves thought and study rather than casting aspersions.
– whuber
Commented Dec 8, 2019 at 21:53
• I agree with that. I wasn't casting aspersions. I just disagreed with your original remark that seemed to suggest that I was missing something in my answer. I appreciated Eric Kernfeld's remark. I still think the OP misinterpreted the meaning of the null & alternative hypotheses for Fisher's combination test. Maybe you didn't misunderstand but your comment seemed to suggest something that you might not have intended. Commented Dec 8, 2019 at 22:10

There are several ways of combining $p$-values and some of them have this property and some do not. This is partly because the problem is not well specified. There has been an extensive simulation study of many of the most-well known methods. The bottom line is that if you want the property of cancellation you can have it but you do not have to.