Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
A meta-analysis includes a bunch of studies, all of which reported a P value greater than 0.05. Is it possible for the overall meta-analysis to report a P value less than 0.05? Under what circumstances?
(I am pretty sure the answer is yes, but I'd like a reference or explanation.)
The results of individual studies may be insignificant but viewed together, the results may be significant.
In theory you can proceed by treating the results $y_i$ of study $i$ like any other random variable.
Let $y_i$ be some random variable (eg. the estimate from study $i$). Then if $y_i$ are independent and $E[y_i]=\mu$, you can consistently estimate the mean with:
$$ \hat{\mu} = \frac{1}{n} \sum_i y_i $$
Adding more assumptions, let $\sigma^2_i$ be the variance of estimate $y_i$. Then you can efficiently estimate $\mu$ with inverse variance weighting:
$$\hat{\mu} = \sum_i w_i y_i \quad \quad w_i = \frac{1 / \sigma^2_i}{\sum_j 1 / \sigma^2_j}$$
In either of these cases, $\hat{\mu}$ may be statistically significant at some confidence level even if the individual estimates are not.
If $E[y_i] \neq \mu$ then the meta-analysis may not converge to $\mu$ (i.e. the mean of the meta-analysis is an inconsistent estimator).
For example, if there's a bias against publishing negative results, this simple meta-analysis may be horribly inconsistent and biased! It would be like estimating the probability that a coin flip lands heads by only observing the flips where it didn't land tails!
$y_i$ and $y_j$ may not be independent. For example, if two studies $i$ and $j$ were based upon the same data, then treating $y_i$ and $y_j$ as independent in the meta-analysis may vastly underestimate the standard errors and overstate statistical significance. Your estimates would still be consistent, but the standard-errors need to reasonably account for cross-correlation in the studies.
Combining (1) and (2) can be especially bad.
For example, the meta-analysis of averaging polls together tends to be more accurate than any individual poll. But averaging polls together is still vulnerable to correlated error. Something that has come up in past elections is that young exit poll workers may tend to interview other young people rather than old people. If all the exit polls make the same error, then you have a bad estimate which you may think is a good estimate (the exit polls are correlated because they use the same approach to conduct exit polls and this approach generates the same error).
Undoubtedly people more familiar with meta-analysis may come up with better examples, more nuanced issues, more sophisticated estimation techniques, etc..., but this gets at some of the most basic theory and some of the bigger problems. If the different studies make independent, random error, then meta-analysis may be incredibly powerful. If the error is systematic across studies (eg. everyone undercounts older voters etc...), then the average of the studies will also be off. If you underestimate how correlated studies are or how correlated errors are, you effectively over estimate your aggregate sample size and underestimate your standard errors.
There are also all kinds of practical issues of consistent definitions etc...
Yes. Suppose you have $N$ p-values from $N$ independent studies.
Fisher's test
(EDIT - in response to @mdewey's useful comment below, it is relevant to distinguish between different meta tests. I spell out the case of another meta test mentioned by mdewey below)
The classical Fisher meta test (see Fisher (1932), "Statistical Methods for Research Workers" ) statistic $$ F=-2\sum_{i=1}^N\ln(p_i) $$ has a $\chi^2_{2N}$ null distribution, as $-2\ln(U)\sim\chi^2_2$ for a uniform r.v. $U$.
Let $\chi^2_{2N}(1-\alpha)$ denote the $(1-\alpha)$-quantile of the null distribution.
Suppose all p-values are equal to $c$, where, possibly, $c>\alpha$. Then, $F=-2N\ln(c)$ and $F>\chi^2_{2N}(1-\alpha)$ when $$c < \exp\left(-\frac{\chi^2_{2N}(1-\alpha)}{2N}\right)$$ For example, for $\alpha=0.05$ and $N=20$, the individual $p$-values only need to be less than
> exp(-qchisq(0.95, df = 40)/40)
[1] 0.2480904
Of course, what the meta statistic tests is "only" the "aggregate" null that all individual nulls are true, which is to be rejected as soon as only one of the $N$ nulls is false.
EDIT:
Here is a plot of the "admissible" p-values against $N$, which confirms that $c$ grows in $N$, although it seems to level off at $c\approx0.36$.
I found an upper bound for the quantiles of the $\chi^2$ distribution $$ \chi^2_{2N}(1-\alpha)\leq 2N+2\log(1/\alpha)+2\sqrt{2N\log(1/\alpha)}, $$ here, suggesting that $\chi^2_{2N}(1-\alpha)=O(N)$ so that $\exp\left(-\frac{\chi^2_{2N}(1-\alpha)}{2N}\right)$ is bounded from above by $\exp(-1)$ as $N\to\infty$. As $\exp(-1)\approx0.3679$, this bound seems reasonably sharp.
Inverse Normal test (Stouffer et al., 1949)
The test statistic is given by $$ Z=\frac{1}{\sqrt{N}}\sum_{i=1}^N\Phi^{-1}(p_i) $$ with $\Phi^{-1}$ the standard normal quantile function. The test rejects for large negative values, viz., if $Z < -1.645$ at $\alpha=0.05$. Hence, for $p_i=c$, $Z=\sqrt{N}\Phi^{-1}(c)$. When $c<0.5$, $\Phi^{-1}(c)<0$ and hence $Z\to_p-\infty$ as $N\to\infty$. If $c\geq0.5$, $Z$ will take values in the acceptance region for any $N$. Hence, a common p-value less than 0.5 is sufficient to produce a rejection of the meta test as $N\to\infty$.
More specifically, $Z < -1.645$ if $c<\Phi(-1.645/\sqrt{N})$, which tends to $\Phi(0)=0.5$ from below as $N\to\infty$.
The answer to this depends on what method you use for combining $p$-values. Other answers have considered some of these but here I focus on one method for which the answer to the original question is no.
The minimum $p$ method, also known as Tippett's method, is usually described in terms of a rejection at the $\alpha_*$ level of the null hypothesis. Define $$ p_{[1]} \le p_{[2]} \dots p_{[k]} $$ for the $k$ studies. Tippett's method then evaluates whether \begin{equation} p_{[1]} < 1 - (1 - \alpha_*)^{\frac{1}{k}} \end{equation}
It is easy to see the since the $k$th root of a number less than unity is closer to unity the last term is greater than $\alpha_*$ and hence the overall result will be non-significant unless $p_{[1]}$ is already less than $\alpha_*$.
It is possible to work out the critical value and for example if we have ten primary studies each with a $p$-values of 00.05 so as close to significant as can be then the overall critical value is 0.40. The method can be seen as a special case of Wilkinson's method which uses $p_{[r]}$ for $1\le r\le k$ and in fact for the particular set of primary studies even $r=2$ is not significant ($p=0.09$)
L H C Tippett's method is described in a book The methods of statistics. 1931 (1st ed) and Wilkinson's method is here in an article "A statistical consideration in psychological research"
