37

You can perfectly use the mean $p$-value. Fisher’s method set sets a threshold $s_\alpha$ on $-2 \sum_{i=1}^n \log p_i$, such that if the null hypothesis $H_0$ : all $p$-values are $\sim U(0,1)$ holds, then $-2 \sum_i \log p_i$ exceeds $s_\alpha$ with probability $\alpha$. $H_0$ is rejected when this happens. Usually one takes $\alpha = 0.05$ and $s_\alpha$...


31

In theory, yes... The results of individual studies may be insignificant but viewed together, the results may be significant. In theory you can proceed by treating the results $y_i$ of study $i$ like any other random variable. Let $y_i$ be some random variable (eg. the estimate from study $i$). Then if $y_i$ are independent and $E[y_i]=\mu$, you can ...


30

Yes. Suppose you have $N$ p-values from $N$ independent studies. Fisher's test (EDIT - in response to @mdewey's useful comment below, it is relevant to distinguish between different meta tests. I spell out the case of another meta test mentioned by mdewey below) The classical Fisher meta test (see Fisher (1932), "Statistical Methods for Research Workers" ...


27

What is wrong with summing up all individual $p$-values? As @whuber and @Glen_b argue in the comments, Fisher's method is essentially multiplying all individual $p$-values, and multiplying probabilities is a more natural thing to do than adding them. Still one can add them up. In fact, precisely this was suggested by Edgington (1972) An additive method for ...


22

I like your question, but unfortunately my answer is NO, it doesn't prove $H_0$. The reason is very simple. How would do you know that the distribution of p-values is uniform? You would probably have to run a test for uniformity which will return you its own p-value, and you end up with the same kind of inference question that you were trying to avoid, only ...


21

Your series of experiments can be viewed as a single experiment with far more data, and as we know, more data is advantageous (eg. typically standard errors decrease as $\sqrt{n}$ increases for independent data). But you ask, "Is this ... enough evidence to conclude that H0 is true?" No. A basic problem is that another theory may produce similar patterns ...


17

One flaw that jumps out is Stouffer's method can detect systematic shifts in the $z_i$, which is what one would usually expect to happen when one alternative is consistently true, whereas the chi-squared method would appear to have less power to do so. A quick simulation shows this to be the case; the chi-squared method is less powerful to detect a one-sided ...


13

The way that Fisher's approach measures the combined effect of p-values is to effectively look at their product (the ordering of possible statistics when adding the logs is the same as when taking the product). It then asks whether this is unusually low compared to what you'd find with random p-values when the null is true (which would be draws from a ...


10

The p-value from each experiment should have a uniform distribution between 0 and 1 under the null hypothesis, so tests of the null hypothesis over all experiments can be based on this. Perhaps the most common test statistic is Fisher's: for p-values $p_j$ from $m$ independent experiments the negative log of each follows an exponential distribution $$-\log ...


10

So if you did three studies of similar sizes and got a p-value of 0.05 on all three occasions, your intuition is that the "true value" should be 0.05? My intuition is different. Multiple similar results would seem to make the significance higher (and therefore the p-values which are probabilities should be lower). P-values are not really probabilities. They ...


10

One general way to gain insight into test statistics is to derive the (usually implicit) underlying assumptions that would lead that test statistic to be most powerful. For this particular case a student and I have recently done this: http://arxiv.org/abs/1111.1210v2 (a revised version is to appear in Annals of Applied Statistics). To very briefly ...


9

i) First, a recommendation: Use pchisq( -2*sum(log(p-values)), df, lower.tail=FALSE) instead of 1- ... - you're likely to end up with more accuracy for small p-values. To see that they're sometimes going to give different results, try this: x=70;c(1-pchisq(x,1),pchisq(x,1,lower.tail=FALSE)) ii) Yes, it's one-sided. Small values of the chi-square ...


8

This is to make the variance of the statistic equal to $1$. The statistic, $S$, is a linear combination of independent $z$-scores: $$ S = \frac{\sum_{i=1}^{k} Z_{i}}{\sqrt{k}} $$ so the variance of $S$ is $$ {\rm var}(S) = {\rm var}\left( \frac{\sum_{i=1}^{k} Z_{i}}{\sqrt{k}} \right) = \frac{1}{k} \sum_{i=1}^{k} {\rm var}( Z_{i}) = \frac{1}{k} \cdot k ...


8

The metap package by Michael Dewey implements many methods for combining p-values: sumlog: Fisher's method sumz: Looks like Stouffer's method (with weights), this isn't mentioned explicitly in the function's documentation but confirmed in the draft vignette (which is not part of the package yet) meanp: When combining p-values, why not just averaging? ...


7

The higher overall sample size leads to a higher power and thus to a smaller p value (at least if the working hypothesis is supported by the data). This is usually the main point of any meta analysis: multiple weak evidences supporting a hypothesis are combined to strong evidence for it.


7

Your p-value looks to be correct. Consider that if the null hypothesis is true, p-values should be uniform; when you have many of them, you're effectively checking your collection of p-values for consistency with uniformity, against the alternative that they're smaller than you'd expect from a uniform (Fisher's method measures this degree of being too small ...


7

If underflow is the issue, you could simply compare the sum of their logs (the mean of the logs is entirely equivalent) Note that Fishers method actually compares $-2\sum_{i=1}^k \log(p_i)$ to a $\chi^2_{2k}$ ... so Fisher would also be working on the log scale. However, it's not clear to me that there's necessarily any particularly meaningful comparison ...


7

It sounds like you've fallen prey to a common misconception about $p$-values. $p$ does not tell you the probability that the null hypothesis is true. Rather, it is the probability, assuming the null hypothesis is true, of observing a test statistic as extreme as the one you've observed. (And when the null hypothesis is false, it has no clear meaning at all.) ...


5

The Fisher combination test is intended to combine information from separate tests done on independent data sets in order to obtain power when the individual tests may not have sufficient power. The idea is that if the $k$ null hypotheses are all correct the $p$-value will be uniformly distributed on $[0,1]$ independently of each other. This means that $-2 ∑...


5

One useful entry into this literature is an annotated bibliography by R D Cousins which considers most of the well-known methods for combining $p$-values and which is freely available from arXiv.


5

There is a whole field of statistics called meta-analysis that deals with this topic. The idea is how to combine the information from different studies. I would not do the mean or the median of the p-values, but there are ways to combine them; but be aware of publication bias, it could be that more studies were done than you know about, but only the ones ...


5

There are a number of methods for combining $p$-values which could be considered. Birnbaum in his paper "Combining independent tests of significance" available here points out the problem is poorly specified. This may account for the number of methods available and their differing behaviour. The null hypothesis $H_0$ is well defined, that all $p_i$ have a ...


5

If you convert $p<0.05$ to $p=0.05$ then your analysis will be conservative but you will at least have been able to include all the studies. Similarly for $p < 0.01$ and so on. The problem comes from the ones which say $p > 0.05$ as the only safe option here is to convert them to $p = 1$, your suggestion of $p = 0.5$ cannot really be justified. Be ...


5

As you have found out Fisher's method does not cancel values in opposite directions. The same is true of Tippett's method (which uses the minimum $p$). However the good news is that Stouffer's method (which $z$-transforms the $p$) and Edgington's method (which sums the $p$) do cancel so you could use one of them instead. If you go to the page for the metap ...


5

In a sense you are right (see the p-curve) with some small caveats: you need the test to have some power under the alternative. Illustration of the potential problem: generating a p-value as a uniform distribution on 0 to 1 and rejecting when $p \leq \alpha$ is a (admittedly pretty useless) level $\alpha$ test for any null hypothesis, but you will get a ...


4

According to Becker's chapter on combining $p$-values in Cooper and Hedges book you can use $$ \frac{\sum {t_f}_i (p_i)}{{\sqrt{\sum\frac{f_i}{f_i-2}}}} > z(\alpha) $$ where ${t_f}_i$ is Student's $t$ with $f_i$ the degrees of freedom $p_i$ the p-value and $\alpha$ is the desired significance value. She does not give a reference for the method which ...


4

To combine p-values means to find formulas $g(p_1,p_2, \ldots, p_n)$ (one for each $n\ge 2$) for which $g$ is symmetric in its arguments; $g$ is strictly increasing separately in each variable; and $P=g(P_1,\ldots, P_n)$ has a uniform distribution when the $P_i$ are independently uniformly distributed. Symmetry means no one of the $n$ tests is favored ...


4

The answer to this depends on what method you use for combining $p$-values. Other answers have considered some of these but here I focus on one method for which the answer to the original question is no. The minimum $p$ method, also known as Tippett's method, is usually described in terms of a rejection at the $\alpha_*$ level of the null hypothesis. Define ...


4

As explained at https://stats.stackexchange.com/a/314739/919, Fisher's Method combines p-values $p_1, p_2, \ldots, p_n$ under the assumption they arise independently under null hypotheses with continuous test statistics. This means each is independently distributed uniformly between $0$ and $1.$ A simple calculation establishes that $-2\log(p_i)$ has a $\...


4

There are many ways to calculate an overall $p$-value: Edgington's method, Fisher's, Lancaster's, Stouffer, Tippett's and several others. They all have their advantages and disadvantages. Since you have tagged this R you may be interested to read the vignettes for the metap package available from CRAN which contains some guidance on the choice. It is too ...


Only top voted, non community-wiki answers of a minimum length are eligible