8
$\begingroup$

I am using Fisher's combined test to fuse several different independent tests. I have a problem understanding the results in some cases.

Example: Let's say I run two different tests, both with the hypothesis that mu is smaller than 0. Let's say that n is identical and the two samples have the same calculated variance. However, let's assume that one test yielded an average that is $1.5$ and the other is $-1.5$. I will get two complementing p-vals (e.g., $0.995$ & $0.005$). Interestingly, combining the two brings about a significant $p$-value in the Fisher test: $p=0.0175$.

This is weird because I could have chosen the exact opposite test $(\mu>0)$ and sampled results - and still get $p=0.0175$. It's almost as if the Fisher test does not take the direction of the hypothesis into account.

Can anyone explain this?

Thanks

$\endgroup$
3
  • 2
    $\begingroup$ If I interpret this question correctly, the discussion in Rice, A Consensus Combined P-Value Test and the Family-wide Significance of Component Tests (Biometrics 1990) explains this problem: see p. 304. The paper offers a solution. $\endgroup$
    – whuber
    Jun 25 '12 at 13:08
  • $\begingroup$ Actually using Fisher's combined probability test the combined p for 0.995 and 0.005 is 0.03. Not that it changes the interpretation (smile) but I am wondering where the 0.0175 came from. $\endgroup$
    – AussieAndy
    Jan 19 '16 at 23:35
  • $\begingroup$ @AussieAndy Yes, I agree -- I make it about 0.03136 $\endgroup$
    – Glen_b
    Jan 20 '16 at 1:10
9
$\begingroup$

The Fisher combination test is intended to combine information from separate tests done on independent data sets in order to obtain power when the individual tests may not have sufficient power. The idea is that if the $k$ null hypotheses are all correct the $p$-value will be uniformly distributed on $[0,1]$ independently of each other. This means that $-2 ∑ \log(p_i)$ will be $\chi^2$ with $2k$ degrees of freedom. Rejecting this combined null hypothesis leads to the conclusion that at least one of the null hypotheses is false. That is what you are doing when you apply this procedure.

$\endgroup$
6
  • 1
    $\begingroup$ This does not seem to address the real issue raised by the question: because the two p-values are symmetrically opposite, and therefore (at least according to some intuition) ought to "cancel," how is it that Fisher's method produces a "significant" result--and which conclusion does it support?? $\endgroup$
    – whuber
    Jan 19 '16 at 23:42
  • 2
    $\begingroup$ That should be $2k$ df. $\endgroup$
    – Glen_b
    Jan 20 '16 at 1:15
  • 2
    $\begingroup$ +1 for Rejecting this combined null hypothesis leads to the conclusion that at least one of the null hypotheses is false. $\endgroup$ Dec 8 '19 at 19:22
  • $\begingroup$ I think the OP & at the time @whuber in his comment are misunderstanding the meaning of the rejection of the combined null hypotheses. eric_kernfield is emphasizing this by repeating what I said in my answer. $\endgroup$ Dec 8 '19 at 19:28
  • $\begingroup$ @Michael, I doubt I misapprehended something as elementary as what it means to reject the combined hypotheses. What is missing from your answer is an explanation of the apparent paradox raised by the OP and in my comment. One place we might seek an explanation is to note that in one case the data were consistent with the null and in the other case they were noticeably inconsistent. The combined dataset thereby still exhibits some inconsistency with the null, which may be why the Fisher p-value is low--but not that low. This deserves thought and study rather than casting aspersions. $\endgroup$
    – whuber
    Dec 8 '19 at 21:53
3
$\begingroup$

There are several ways of combining $p$-values and some of them have this property and some do not. This is partly because the problem is not well specified. There has been an extensive simulation study of many of the most-well known methods. The bottom line is that if you want the property of cancellation you can have it but you do not have to.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.