A meta-analysis includes a bunch of studies, all of which reported a P value greater than 0.05. Is it possible for the overall meta-analysis to report a P value less than 0.05? Under what circumstances?

(I am pretty sure the answer is yes, but I'd like a reference or explanation.)

  • 1
    $\begingroup$ I don't know much about meta-analysis, but I was under the impression that it doesn't involve any hypothesis tests, just an estimate of the population effect, in which case there's no notion of significance to speak of. $\endgroup$ – Kodiologist Oct 28 '16 at 18:55
  • 1
    $\begingroup$ Well, a meta-analysis –at the end of the day– is just a weighted mean. And you can certainly set up an hypothesis test for that weighted mean. See, for example, Borenstein, Michael, et al. "A basic introduction to fixed‐effect and random‐effects models for meta‐analysis." Research Synthesis Methods 1.2 (2010): 97-111. $\endgroup$ – boscovich Oct 28 '16 at 19:07
  • 1
    $\begingroup$ The other answers are also good, but a simple case: two studies are significant at p=0.9 but not p=0.95. The probability that two independent studies would both show p>=0.9 is only 0.01, so your meta analysis could show significance at p = 0.99 $\endgroup$ – user1566 Oct 28 '16 at 20:30
  • 2
    $\begingroup$ Take the limit: No one measurement can provide enough evidence for/against a (nontrivial) hypothesis to have a small $p$ value, but a large enough collection of measurements can. $\endgroup$ – Eric Towers Oct 28 '16 at 23:02
  • $\begingroup$ p- values do not indicate either "statistically significant" or insignificant effect. What could we understand from a significant conclusion? Is it a meta analytic conclusion ? $\endgroup$ – Subhash C. Davar May 11 '18 at 5:43

In theory, yes...

The results of individual studies may be insignificant but viewed together, the results may be significant.

In theory you can proceed by treating the results $y_i$ of study $i$ like any other random variable.

Let $y_i$ be some random variable (eg. the estimate from study $i$). Then if $y_i$ are independent and $E[y_i]=\mu$, you can consistently estimate the mean with:

$$ \hat{\mu} = \frac{1}{n} \sum_i y_i $$

Adding more assumptions, let $\sigma^2_i$ be the variance of estimate $y_i$. Then you can efficiently estimate $\mu$ with inverse variance weighting:

$$\hat{\mu} = \sum_i w_i y_i \quad \quad w_i = \frac{1 / \sigma^2_i}{\sum_j 1 / \sigma^2_j}$$

In either of these cases, $\hat{\mu}$ may be statistically significant at some confidence level even if the individual estimates are not.

BUT there may be big problems, issues to be cognizant of...

  1. If $E[y_i] \neq \mu$ then the meta-analysis may not converge to $\mu$ (i.e. the mean of the meta-analysis is an inconsistent estimator).

    For example, if there's a bias against publishing negative results, this simple meta-analysis may be horribly inconsistent and biased! It would be like estimating the probability that a coin flip lands heads by only observing the flips where it didn't land tails!

  2. $y_i$ and $y_j$ may not be independent. For example, if two studies $i$ and $j$ were based upon the same data, then treating $y_i$ and $y_j$ as independent in the meta-analysis may vastly underestimate the standard errors and overstate statistical significance. Your estimates would still be consistent, but the standard-errors need to reasonably account for cross-correlation in the studies.

  3. Combining (1) and (2) can be especially bad.

    For example, the meta-analysis of averaging polls together tends to be more accurate than any individual poll. But averaging polls together is still vulnerable to correlated error. Something that has come up in past elections is that young exit poll workers may tend to interview other young people rather than old people. If all the exit polls make the same error, then you have a bad estimate which you may think is a good estimate (the exit polls are correlated because they use the same approach to conduct exit polls and this approach generates the same error).

Undoubtedly people more familiar with meta-analysis may come up with better examples, more nuanced issues, more sophisticated estimation techniques, etc..., but this gets at some of the most basic theory and some of the bigger problems. If the different studies make independent, random error, then meta-analysis may be incredibly powerful. If the error is systematic across studies (eg. everyone undercounts older voters etc...), then the average of the studies will also be off. If you underestimate how correlated studies are or how correlated errors are, you effectively over estimate your aggregate sample size and underestimate your standard errors.

There are also all kinds of practical issues of consistent definitions etc...

| cite | improve this answer | |
  • 1
    $\begingroup$ I'm criticizing a meta-analysis for ignoring dependencies between effect sizes (i.e., many effect sizes were based on the same participants, but treated as independent). The authors say no biggie, we are just interested in moderators anyways. I'm making the point you made here: treating them "as independent in the meta-analysis may vastly underestimate the standard errors and overstate statistical significance." Is there a proof/simulation study showing why this is the case? I have lots of references saying that correlated errors means underestimated SE... but I don't know why? $\endgroup$ – Mark White Jul 4 '17 at 5:58
  • 1
    $\begingroup$ @MarkWhite The basic idea isn't more complicated than $\operatorname{Var}\left( \frac{1}{n} \sum_i X_i \right) = \frac{1}{n^2} \left( \sum_{i} \operatorname{Var}(X_i) + \sum_{i \neq j} \operatorname{Cov}(X_i, X_j) \right)$. If for all $i$ we have $\operatorname{Var}(X_i) = \sigma^2$ and $\operatorname{Cov}(X_i, X_j) = 0$ for $i\neq j$ then $\operatorname{Var}\left( \frac{1}{n} \sum_i X_i \right) = \frac{\sigma^2}{n}$ and your standard error is $\frac{\sigma}{\sqrt{n}}$. On the other hand, if the covariance terms are positive and big, the standard error is going to be larger. $\endgroup$ – Matthew Gunn Jul 4 '17 at 15:01
  • $\begingroup$ @MarkWhite I'm not a meta-analysis expert, and I honestly don't know what's a great source for how one should do modern, meta-analysis. Conceptually, replicating analysis on the same data is certainly useful (as is intensively studying some subjects), but it's not the same as reproducing a finding on new, independent subjects. $\endgroup$ – Matthew Gunn Jul 4 '17 at 15:44
  • 1
    $\begingroup$ Ah, so in words: The total variance of an effect size comes from (a) its variance and (b) it's covariance with other effect sizes. If the covariance is 0, then standard error estimate is fine; but if it covaries with other effect sizes, we need to account for that variance, and ignoring it means we are underestimating the variance. It's like variance is made up of two parts A and B, and ignoring dependencies assumes the B part is 0 when it is not? $\endgroup$ – Mark White Jul 4 '17 at 16:01
  • 1
    $\begingroup$ Also, this looks to be a good source (see especially Box 2): nature.com/neuro/journal/v17/n4/pdf/nn.3648.pdf $\endgroup$ – Mark White Jul 4 '17 at 16:03

Yes. Suppose you have $N$ p-values from $N$ independent studies.

Fisher's test

(EDIT - in response to @mdewey's useful comment below, it is relevant to distinguish between different meta tests. I spell out the case of another meta test mentioned by mdewey below)

The classical Fisher meta test (see Fisher (1932), "Statistical Methods for Research Workers" ) statistic $$ F=-2\sum_{i=1}^N\ln(p_i) $$ has a $\chi^2_{2N}$ null distribution, as $-2\ln(U)\sim\chi^2_2$ for a uniform r.v. $U$.

Let $\chi^2_{2N}(1-\alpha)$ denote the $(1-\alpha)$-quantile of the null distribution.

Suppose all p-values are equal to $c$, where, possibly, $c>\alpha$. Then, $F=-2N\ln(c)$ and $F>\chi^2_{2N}(1-\alpha)$ when $$c < \exp\left(-\frac{\chi^2_{2N}(1-\alpha)}{2N}\right)$$ For example, for $\alpha=0.05$ and $N=20$, the individual $p$-values only need to be less than

> exp(-qchisq(0.95, df = 40)/40)
[1] 0.2480904

Of course, what the meta statistic tests is "only" the "aggregate" null that all individual nulls are true, which is to be rejected as soon as only one of the $N$ nulls is false.


Here is a plot of the "admissible" p-values against $N$, which confirms that $c$ grows in $N$, although it seems to level off at $c\approx0.36$.

enter image description here

I found an upper bound for the quantiles of the $\chi^2$ distribution $$ \chi^2_{2N}(1-\alpha)\leq 2N+2\log(1/\alpha)+2\sqrt{2N\log(1/\alpha)}, $$ here, suggesting that $\chi^2_{2N}(1-\alpha)=O(N)$ so that $\exp\left(-\frac{\chi^2_{2N}(1-\alpha)}{2N}\right)$ is bounded from above by $\exp(-1)$ as $N\to\infty$. As $\exp(-1)\approx0.3679$, this bound seems reasonably sharp.

Inverse Normal test (Stouffer et al., 1949)

The test statistic is given by $$ Z=\frac{1}{\sqrt{N}}\sum_{i=1}^N\Phi^{-1}(p_i) $$ with $\Phi^{-1}$ the standard normal quantile function. The test rejects for large negative values, viz., if $Z < -1.645$ at $\alpha=0.05$. Hence, for $p_i=c$, $Z=\sqrt{N}\Phi^{-1}(c)$. When $c<0.5$, $\Phi^{-1}(c)<0$ and hence $Z\to_p-\infty$ as $N\to\infty$. If $c\geq0.5$, $Z$ will take values in the acceptance region for any $N$. Hence, a common p-value less than 0.5 is sufficient to produce a rejection of the meta test as $N\to\infty$.

More specifically, $Z < -1.645$ if $c<\Phi(-1.645/\sqrt{N})$, which tends to $\Phi(0)=0.5$ from below as $N\to\infty$.

| cite | improve this answer | |
  • 2
    $\begingroup$ +1 and wow! did not expect there to be an upper bound at all, let alone $1/e$. $\endgroup$ – amoeba Oct 29 '16 at 9:56
  • $\begingroup$ Thanks :-). I had not expected one either before I saw the plot... $\endgroup$ – Christoph Hanck Oct 29 '16 at 10:47
  • 5
    $\begingroup$ Interestingly the method due to Fisher is the only one of the commonly used methods which has this property. For most of the others what you call F increases with N if $c>0.5) and decreases otherwise. That applies to Stouffer's method and Edgington's method as well as methods based on logits and on mean of p. The various methods which are special cases of Wilkinson's method (minimum p, maximum p, etc) have different properties again. $\endgroup$ – mdewey Oct 29 '16 at 12:17
  • 1
    $\begingroup$ @mdewey, that is interesting indeed, I just picked Fisher's test purely because it came to my mind first. That said, by "only one", do you mean the specific bound $1/e$? Your comments, that I try to spell out in my edit, suggest to me that Stouffer's method also has an upper bound, that turns out to be 0.5? $\endgroup$ – Christoph Hanck Oct 30 '16 at 9:15
  • $\begingroup$ I am not going to have time to go into this for another week but I think if you have ten studies with $p=0.9$ you get an overall $p$ as close to unity as makes no difference. There may be a one- versus two-sided issue here. If you want to look at more material I have a draft of extra stuff to go into my R package <code>metap</code> here which you are free to use to expand your answer if you wish. $\endgroup$ – mdewey Oct 30 '16 at 13:02

The answer to this depends on what method you use for combining $p$-values. Other answers have considered some of these but here I focus on one method for which the answer to the original question is no.

The minimum $p$ method, also known as Tippett's method, is usually described in terms of a rejection at the $\alpha_*$ level of the null hypothesis. Define $$ p_{[1]} \le p_{[2]} \dots p_{[k]} $$ for the $k$ studies. Tippett's method then evaluates whether \begin{equation} p_{[1]} < 1 - (1 - \alpha_*)^{\frac{1}{k}} \end{equation}

It is easy to see the since the $k$th root of a number less than unity is closer to unity the last term is greater than $\alpha_*$ and hence the overall result will be non-significant unless $p_{[1]}$ is already less than $\alpha_*$.

It is possible to work out the critical value and for example if we have ten primary studies each with a $p$-values of 00.05 so as close to significant as can be then the overall critical value is 0.40. The method can be seen as a special case of Wilkinson's method which uses $p_{[r]}$ for $1\le r\le k$ and in fact for the particular set of primary studies even $r=2$ is not significant ($p=0.09$)

L H C Tippett's method is described in a book The methods of statistics. 1931 (1st ed) and Wilkinson's method is here in an article "A statistical consideration in psychological research"

| cite | improve this answer | |
  • 1
    $\begingroup$ Thanks. But note that most meta-analysis methods combine effect sizes (accounting for any difference in sample size), and do not combine P values. $\endgroup$ – Harvey Motulsky Nov 9 '16 at 16:16
  • $\begingroup$ @HarveyMotulsky agreed, combining p-values is a last resort but the OP did tag his question with the combining-p-values tag so I responded in that spirit $\endgroup$ – mdewey Nov 9 '16 at 16:34
  • $\begingroup$ I think that your answer is correct. $\endgroup$ – Subhash C. Davar Apr 27 '18 at 2:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.