What are the reasonable properties of p-value combination methods when all p-values are the same?

Question

Suppose we are testing a null hypothesis $H_0$, by combining evidence from $n$ independent tests, using a p-value combination method $M$. The combined p-value is $M(p_1,...,p_n)$. Assume the extreme case when we get the same p-value from all individual tests, which of the following properties of $M$ are more reasonable:

1. $M_n(p,...,p)\leq p$;
2. $M_n(p,...,p)\geq p$;
3. $M_n(p,...,p)$ decreases with $n$;
4. $M_n(p,...,p)$ increases with $n$;
5. all of the above depends on the value of $p$.

Take the Fisher's combination method as an example: $$M_n(p,...,p)=P\{\chi^2_{2n}>-2n \ln(p)\}\approx P\{Z>-\sqrt{2n}\ln(p)-\sqrt{2n}\}.$$The combined p-value decreases when $p<e^{-1}=0.367$.

At the same time, with the minimum $p$-value method, $$M_n(p,...,p)=P\{U_{(1)} < p\}=1-(1-p)^n,$$ which always increases with $n$.

In this special case of $p_i=p$, which method is more "reasonable"?

Answer 1

Interestingly most of the theoretical work on meta-analysis of significance values is quite old. One important source is an article by Birnbaum, 1954, Combining Independent Tests of Significance and I think his abstract is worth quoting in full.

It is shown that no single method of combining independent tests of significance is optimal in general, and hence that the kinds of tests to be combined should be considered in selecting a method of combination. A number of proposed methods of combination are applied to a particular common testing problem. It is shown that for such problems Fisher's method and a method proposed by Tippett have an optimal property.

Those two methods are of course the two you mention.

Two other sources which might shed some light are a paper by Cousins in ArXiv (Annotated Bibliography of Some Papers on Combining Significances or p-values, 2007) and a simulation study by Loughin (Annotated Bibliography of Some Papers on Combining Significances or p-values, 2004). There is also a 1958 paper which I have not read by T Lipták called On the combination of independent tests in a Hungarian journal which examines the problem from an even more general angle or so I understand from secondary sources.