4
$\begingroup$

Why is Stouffer's method so often performed on $z$'s that correspond to one-tailed $p$-values, when the mathematics allows for $z$'s that correspond to two-tailed $p$-values?

$\endgroup$
  • 2
    $\begingroup$ Joel, I've noticed you've asked several questions (and gotten several answers with upvotes) but never accepted an answer. Consider upvoting and/or accepting answers you've found helpful. $\endgroup$ – Macro Jan 17 '12 at 14:45
  • $\begingroup$ Can you clarify your question a little bit, maybe even providing an example (or a reference to one) that you're thinking of? In particular, it's not clear what you mean by "one-tailed $p$-values" and "two-tailed $p$-values" here since Stouffer's method may be combining $p$-values for tests that have nothing to do with $t$-tests. $\endgroup$ – cardinal Jan 17 '12 at 15:09
  • $\begingroup$ @Joel: I was wondering if you might be interested in an alternate answer to your question. My (perhaps overly earnest) hope was that the current answer would be updated, but that doesn't appear to have happened yet. $\endgroup$ – cardinal Feb 5 '12 at 19:42
-3
$\begingroup$

Suppose the null hypothesis $\mu = 0$ is considered in two $2$-tailed studies. Suppose that one study rejects the null hypothesis because all data are strongly positive (supporting the alternative hypothesis $\mu > 0$ as well as the alternative hypothesis $\mu \neq 0$), while the other study rejects the null because all the data are strongly negative (supporting the alternative hypothesis $\mu < 0$ as well as the alternative hypothesis $\mu \neq 0$). Clearly, if the data from the two studies were combined, the null hypothesis would be rejected because all the data differ significantly from $0$ and thus support the alternative hypothesis $\mu \neq 0$ corresponding to a $2$-tailed study. However, the $z$'s from the studies will be positive and negative respectively, and Stouffer's method will add the two $z$ scores to get an answer close to $0$ and thus say that the null hypothesis should not be rejected at all.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Maybe I am misunderstanding your answer then which discusses two-tailed studies which I interpreted as using a $p$-value from a two-tailed test which would involve the absolute value of the respective means. $\endgroup$ – cardinal Jan 17 '12 at 16:16
  • 1
    $\begingroup$ First of all, if we're discussing two-tailed tests, then our $p$-value is based on $p = \mathbb P(|T| > t_{1-\alpha/2})$, say. Hence, anything far away from zero will give a small $p$-value. The $p$-value encodes nothing about the sign of $T$. Thus, when "backtransforming" this $p$-value in the Stouffer's test, we would get a large value of $Z_i = \Phi^{-1}(1-p_i)$ regardless of the sign of $T_i$. .../... $\endgroup$ – cardinal Jan 17 '12 at 16:47
  • 1
    $\begingroup$ Maybe it boils down to what is meant by $z$ and how it relates to $p$. You are using absolute values; I am not. If Stouffer's method sums absolute values, then no problem; if it sums signed values, then there is a problem as indicated in my answer. In meta-analysis of one-sided tests $\mu \leq 0$ (or the traditional $\mu = 0$) versus $\mu > 0$, $z$ is always positive and so maybe nobody mentions that absolute values are to be summed because there is no need to. If so, one should not blindly apply Stouffer's method in $2$-sided tests and sum $z$ scores if $z$ scores can be positive or negative. $\endgroup$ – Dilip Sarwate Jan 17 '12 at 17:06
  • 2
    $\begingroup$ I think @cardinal is correct Dilip. Most of your reply looks good, but this part seems mistaken: "However, the z's from the studies will be positive and negative respectively." This presumes the z's would be calculated as if the studies performed one -tailed tests. With a correct two-tailed test, the z's will both be positive, not "positive and negative respectively" as you claim. There's still some thinking to do here: although Stouffer's method will conclude there is a significant difference, it cannot tell us in what direction! $\endgroup$ – whuber Jan 17 '12 at 19:46
  • 1
    $\begingroup$ I have concern with the first part of the answer : in one sample, we have most $z_i\ll0$, in a second sample, most $z_i\gg0$. Then "if the data from the two studies were combined, the null hypothesis would be rejected because all the data differ significantly from 0 and thus support the alternative hypothesis $\mu\ne0$". I don’t agree. First sample $-65.8,-82.5,-1.6,-19.9,-95.0,-83.1$, two sided $p=0.01$; second sample $37.6,43.3,56.1,66.6,58.8,89.2,79.2,77.2,144.3$, $p=0.0001$; pooled data, $p=0.3$. (data generated with rnorm) $\endgroup$ – Elvis Jan 17 '12 at 20:17

Not the answer you're looking for? Browse other questions tagged or ask your own question.