13
$\begingroup$

It seems like a fairly straightforward question, but when I really think about it, Stouffer's method doesn't make sense to me. This is why:

Assume a two-tailed hypothesis. You first calculate $z_i$ from $p$-values. So let's take a fairly simple example. Let's take two $p$-values of $0.05$. This means that $z_1$ and $z_2$ are both $\approx1.96$. According to Stouffer's method, $z_1$ and $z_2$ are combined such that:

$$ Z = \frac{\sum\limits_{i=1}^kZ_i}{\sqrt{k}} = \frac{1.96 + 1.96}{\sqrt{2}} = 2.77 $$

This $z$-score then gets converted to a $p$-value once again, resulting in a $p$-value of $0.005$, whereas the $p$-values from each $z_i$ individually is about $0.05$.

In this sense, it seems as though Stouffer's test artificially changes the resultant $p$-value to a value dissimilar to the $p$-values of each $z_i$, which to me, doesn't make sense.

Am I misunderstanding this test or can someone help me understand how / why it works?

Improve this question
$\endgroup$
12
  • 8
    $\begingroup$ (+1) But please note that Stouffer's method in this form is not appropriate for two-tailed alternatives. The problem is that it overlooks the possibility that one study might have found an effect in one direction and the other, an effect in the opposite direction. One has to check that this has not occurred. To get to your question: in what sense is this "artificial"? Bear in mind the purpose is to combine evidence to support decision making. Doesn't it make sense that two significant results ought to constitute stronger support for a decision than either one alone? $\endgroup$
    – whuber ♦
    Commented Jul 28, 2015 at 20:08
  • $\begingroup$ When I wrote that it seems "artificial," I meant that in the case that there are two samples (N = 2), there will always be an inflation in the Z-score, resulting in consistently lower p-values than expected from either z-score ($z_i$). While it does make sense that two significant results should result in a stronger support for a decision than either one alone, it doesn't make sense for two p-values to be implemented into Stouffer's method and the result be completely different from either p-value. $\endgroup$
    – will
    Commented Jul 28, 2015 at 20:17
  • 2
    $\begingroup$ @will, I cannot understand the last sentence of your first (long) comment here. Yes, it does make sense that two significant results yield a stronger support when combined. Which means that the combined p-value can well be lower than either of the two. So what's the problem? $\endgroup$
    – amoeba
    Commented Jul 28, 2015 at 20:40
  • 3
    $\begingroup$ I was thinking that one way to develop your intuition would be to reverse this procedure: take a single study and split it into two random parts, then analyze each part separately. As a very simple example, consider a post-election survey in which 1000 people were polled and 535 said they voted for the incumbent and 465 for her opponent. A random split might go 265-235 in one half and 270-230 in the other half. What are the p-values for the test of equality of proportions in the two halves and what is the p-value overall? (In R, compute using prop.test(535,1000), etc.) $\endgroup$
    – whuber ♦
    Commented Jul 28, 2015 at 21:44
  • 2
    $\begingroup$ You seem to confuse the sample estimate of the proportion with the p-value of the test!! The overall p-value is 0.03 while the p-values of the two halves are 0.08 and 0.19. $\endgroup$
    – whuber ♦
    Commented Oct 15, 2018 at 18:51

4 Answers 4

Reset to default
9
$\begingroup$

The higher overall sample size leads to a higher power and thus to a smaller p value (at least if the working hypothesis is supported by the data).

This is usually the main point of any meta analysis: multiple weak evidences supporting a hypothesis are combined to strong evidence for it.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Since the statistical term "power" in this context has a sharply different meaning than p-value, I am concerned that this explanation could cause some confusion among them. $\endgroup$
    – whuber ♦
    Commented Jul 28, 2015 at 20:10
  • $\begingroup$ So does this mean that in the case that the sample size is 2, the power of Stouffer's method will always lowered, and that the p-value will always be smaller? How can get a more accurate answer when the sample size is two? $\endgroup$
    – will
    Commented Jul 28, 2015 at 20:19
  • $\begingroup$ The "meta sample" size is two, i.e. there were two experiments who both yielded $p=0.05$. The combined sample size $N$ is $N=N_1+N_2$, so typically much larger than 2. Since this meta analysis only takes into account the p values, the information available is much lower than from the raw data of the $N_1+N_2$ events. $\endgroup$
    – quazgar
    Commented Oct 15, 2018 at 17:44
2
$\begingroup$

For simplicity think in terms of a test on means. Suppose under H0 the treatment effect is zero, so that each z value is a weighted estimate of the treatment effect θi. Stouffer's method gives an unweighted average of these treatment effects so will give a more precise estimate (and hence smaller p-value) than each separate z value. This unweighted estimate of the treatment effect is biased but a weighted Stouffer's method is possible, and if the weights are proportional to 1/standard error(θi) the treatment effect estimate is unbiased. This only makes sense however if the separate z values are measures of the same quantity. An advantage of Stouffer's and Fisher's methods is that they can also be applied to meta-analyses where different response variables have been chosen - so they can't be averaged - but where a consistency of direction of effect can still be discerned.

Improve this answer
$\endgroup$
0
$\begingroup$

Think of it from a meta-analysis point of view: If there was no effect ($H_0$), $p$ values would be equally distributed between 0 and 1. So if you get $p<0.1$ in more than 10% of all single analyses (potentially many of them), this can amount to the conclusion that $H_0$ probably should be rejected.

I do not even see a problem for two-tailed tests: In this case the result should be interpreted as: It is unlikely that the true mean is 0 (in the example of a gaussian around 0), but I cannot tell (from either the previous or the combined $p$ value) if the true mean is above or below it.

Improve this answer
$\endgroup$
-2
$\begingroup$

I think it'd be fine to combine 2-tailed results because that means that the result would amount to zero (if there is evidence that the treatment enhances [right-tail] the disease of a patient but also evidence that it worsens [left-tail], the net result is no evidence towards a particular hypothesis since they cancel out and more observations are needed.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ I do not think that this addresses the question. Also, whuber's comment indicates that this particular method does not work for 2-tailed tests. $\endgroup$
    – mkt
    Commented Nov 19, 2017 at 22:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.