I am a bit confused about doing the meta analysis with my data. Given z-scores I calculate p values in a way: p.values = 2*pnorm(-abs(z)). Then I would like to apply Fischer/or Stouffer methods to do the meta analysis. Those methods work with one-sided p-values, so I just need to divide my p.values by 2 (p.values/2) and perform combine.test(p.values/2) in R?

In this paper I found following: After combining the P-values, if desired the resulting combined P can be again converted to a two-tailed test by multiplying it by two.

So, after I get the combined p values, i multiply them by two to get back to the two-tailed test.

Would it be a correct procedure?


I found this post, saying: Specifically, if two-sided p-values are being analyzed, the two-sided p-value (${p}_i$/2) is used, or 1-${p}_i$ if left-tailed p-values are used.

but I do not understand what is ${p}_i$ in this case. Is ${p}_i$ a two-sided p-value?

  • $\begingroup$ Isn't this a non-standard and potentially misleading way to conduct meta-analysis? I would expect to see an analysis in terms of effect size. $\endgroup$ – rolando2 Jan 8 '16 at 0:27

What is important here is that the $p$-values $p_1,\ldots,p_n$ are independent and constructed so that if the null $H_i$ is true, $p_i\sim U[0,1]$. They are not restricted to one-sided tests/p-values.

For example, for Fisher's test with test statistic $$F=-2\sum_i\ln(p_i),$$ we obtain its null distribution ($\chi^2_{2n}$) as follows: let $y=g(p_i):=-2\ln(p_i)$. Then, $p_i=g^{-1}(y)=e^{-\frac{1}{2}y}$ and the density of $-2\ln(p_i)$ is given by
$$ f_{-2\ln(p_i)}(y)=f_{p_i}(g^{-1}(y))\left|\frac{\partial}{\partial y}g^{-1}(y)\right|. $$ Note $$\frac{\partial}{\partial y}g^{-1}(y)=-\frac{1}{2}e^{-\frac{1}{2}y}$$ and $$\left|\frac{\partial}{\partial y}g^{-1}(y)\right|=\frac{1}{2}e^{-\frac{1}{2}y}.$$ We have $f_{p_i}(g^{-1}(y))=1\;\forall\;g^{-1}(y)\in[0,1]$ (a standard uniform density). This implies $$f_{-2\ln(p_i)}(y)=\frac{1}{2}e^{-\frac{1}{2}y}.$$ The density of a $\chi^2_R$ random variable is $$f_{\chi^2_R}(y)=\frac{1}{2^{R/2}\Gamma(R/2)}y^{\frac{R}{2}-1}e^{-\frac{y}{2}}.$$ With $R=2$, we get $f_{\chi^2_2}(y)=\frac{1}{2\Gamma(1)}e^{-\frac{y}{2}}.$ Recall that $\Gamma(1)=\int_0^\infty t^{1-1}e^{-t}\;dt=1$. So, $$ f_{\chi^2_2}(y)=\frac{1}{2}e^{-\frac{y}{2}}. $$ We have shown that $f_{-2\ln(p_i)}(y)=f_{\chi^2_2}(y)$. The proof is complete since the sum of $n$ independent $\chi^2_R$ r.v.s is distributed as $\chi^2_{n\cdot R}$.

That and why $p$-values are uniform under the null is for example discussed here: Why are p-values uniformly distributed under the null hypothesis?

My conjecture for your mistaken claim that you may only use one-sided p-values might be addressed here: Fisher's method for combing p-values - what about the lower tail?

The link explains that only large values of the test statistic provides evidence against the meta-null that all nulls $H_i$ are true, because the way the statistic is constructed, small $p_i$ translate into a large test statistic $F$.

  • $\begingroup$ So, it means that I could also use two-sided p-values for Fisher or Stouffer, doesn't it? The only thing is that, if I combine two-sided p-values I am more strict regarding the hypothesis. $\endgroup$ – Anni Jan 7 '16 at 19:31
  • 1
    $\begingroup$ For the first question, yes. For the second claim, yes in the sense that if your observed test statistic is, say, in the right tail of the null distribution, it needs to be further out there if you conduct a two-sided test than if you conducted a right-tailed test. (If your test statistic has is standard normal under the null, a test statistic larger than 1.645 would suffice in the right-tailed case, where it would need to exceed 1.96 in the two-sided case, both at the 5% level.) $\endgroup$ – Christoph Hanck Jan 7 '16 at 19:34
  • $\begingroup$ Essentially, you should do a right-tailed test if only large positive values of the test statistic provide evidence against the null. $\endgroup$ – Christoph Hanck Jan 7 '16 at 19:35

You need to think carefully about your scientific question. If you use two-sided p-values then what do you want to happen if five are significant in one direction and five in the other? Do you want them to cancel out?

In fact you also need to be careful with one-sided p-values as well because different methods of combining them do, or do not, cancel them out. Try your proposed method with (0.0001, 0.0001, 0.9999, 0.9999) and see if it corrsesponds to your scientific question.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.