Suppose we are testing a null hypothesis $H_0$, by combining evidence from $n$ independent tests, using a p-value combination method $M$. The combined p-value is $M(p_1,...,p_n)$. Assume the extreme case when we get the same p-value from all individual tests, which of the following properties of $M$ are more reasonable:

  1. $M_n(p,...,p)\leq p$;
  2. $M_n(p,...,p)\geq p$;
  3. $M_n(p,...,p)$ decreases with $n$;
  4. $M_n(p,...,p)$ increases with $n$;
  5. all of the above depends on the value of $p$.

Take the Fisher's combination method as an example: $$M_n(p,...,p)=P\{\chi^2_{2n}>-2n \ln(p)\}\approx P\{Z>-\sqrt{2n}\ln(p)-\sqrt{2n}\}.$$The combined p-value decreases when $p<e^{-1}=0.367$.

At the same time, with the minimum $p$-value method, $$M_n(p,...,p)=P\{U_{(1)} < p\}=1-(1-p)^n,$$ which always increases with $n$.

In this special case of $p_i=p$, which method is more "reasonable"?

  • 2
    $\begingroup$ Is this a verbatim copy of a question from some test or assignment? $\endgroup$ – amoeba Aug 11 '15 at 11:01
  • $\begingroup$ Of course it's not $\endgroup$ – yliueagle Aug 11 '15 at 12:36
  • $\begingroup$ Okay, sorry for suspicion. +1. $\endgroup$ – amoeba Aug 11 '15 at 12:48
  • $\begingroup$ @Amoeba It would nevertheless make a great question to ask of anyone who is learning about p-values. One could add some other interesting possibilities, such as (6) it depends on the particular test; (7) it depends on the null and alternative hypotheses; and (8) it depends on whether the test is two-sided or one-sided. $\endgroup$ – whuber Aug 11 '15 at 14:57

Interestingly most of the theoretical work on meta-analysis of significance values is quite old. One important source is an article by Birnbaum, 1954, Combining Independent Tests of Significance and I think his abstract is worth quoting in full.

It is shown that no single method of combining independent tests of significance is optimal in general, and hence that the kinds of tests to be combined should be considered in selecting a method of combination. A number of proposed methods of combination are applied to a particular common testing problem. It is shown that for such problems Fisher's method and a method proposed by Tippett have an optimal property.

Those two methods are of course the two you mention.

Two other sources which might shed some light are a paper by Cousins in ArXiv (Annotated Bibliography of Some Papers on Combining Significances or p-values, 2007) and a simulation study by Loughin (Annotated Bibliography of Some Papers on Combining Significances or p-values, 2004). There is also a 1958 paper which I have not read by T Lipták called On the combination of independent tests in a Hungarian journal which examines the problem from an even more general angle or so I understand from secondary sources.

  • $\begingroup$ +1. I took the liberty to insert paper titles and to format the link. $\endgroup$ – amoeba Sep 4 '16 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.