5
$\begingroup$

I have a bunch of independent $p$-values and now I want to combine them using the Fisher's method. However, in R, when a $p$-value is zero, the log(p-value) becomes $-\infty$, so that the test statistic $X^2$ is $\infty$. This will give the Fisher's method $p$-value of 0.

If I check the $p$-values' histogram, they look pretty uniform, so I expect that the Fisher's method should give me a not-so-small $p$-value (should not reject the null). Is there a way to get around this issue? How about the Stouffer's method? Will that method automatically take care of the issue when $p = 0$ (or close to 0)?

Update: I tried the Stouffer's method, but still the issue of $\infty$ is not solved...

$\endgroup$
  • $\begingroup$ Can you provide information on how you are calculating p-values? $\endgroup$ – Sameer May 9 '13 at 3:14
  • 3
    $\begingroup$ Isn't the p-value just less than some number, which is based on the number of simulations you did? E.g. if you did 1000 sims, the p-value given is 0, but it's just <0.001. So you treat it as 0.001. (Or whatever). $\endgroup$ – Jeremy Miles May 9 '13 at 3:34
  • 6
    $\begingroup$ @JeremyMiles When calculating bootstrap or Monte Carlo p-values if no test statistics as extreme as the observed statistic is found amongst the $B$ replicates, one generally sets the p-value to $\frac{1}{B}$ because the test statistic has been observed once (your observed test statistic!). However, this still doesn't account for the fact that you actually have a censored distribution i.e. you cannot observe p-values less than $\frac{1}{B}$. A quick Google search did not lead me anywhere; I wonder if someone else has thought about this problem. $\endgroup$ – Sameer May 9 '13 at 3:58
  • 1
    $\begingroup$ @alittleboy re: "when r=N". According to your equation, when $r$ and $N$ are equal, $p=1$. Doing a 2-sided test doesn't mean calculating $p$, then calculating $1-p$, here, does it? $\endgroup$ – ndoogan May 9 '13 at 4:15
  • 1
    $\begingroup$ @alittleboy, what are "r" & "N" here? You should not have any true 0's, even if you are doing a 2-sided test. $\endgroup$ – gung - Reinstate Monica Aug 12 '15 at 23:21
1
$\begingroup$

Irrespective of the discussion in the comments about how these $p$-values of $0$ arose there are methods for combining $p$-values which can be calculated if $p=0$.

As the OP indicated neither Fisher's method nor Stouffer's works.

The method of Edgington based on the sum of $p$, the closely related mean $p$ method, the method using logit of $p$, Tippett's method based on the minimum $p$ and variants of Wilkinson's method of which Tippett is a special case can all be calculated. Whether that is a sensible thing to do depends on the scientific question of course.

All the methods mentioned are available in the R package metap which, disclaimer, I wrote and maintain.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.