3
$\begingroup$

I have 30 p-values all very small (varying between $10^{-140}$ and $10^{-110}$), and I want to combine them in some way to get a single statistic. I learned about Fisher's method, but if I apply it to those small p-values, what I get is 0, probably due to lost of precision. But I want to get a certain number, not 0, since my purpose is to do the same for multiple methods and compare the statistics for each of them (each of the methods give 30 p-values, again all very small). I get 0 for each of the the methods, although they are in fact different, and getting 0 for all of the methods makes it impossible to compare them. A way is to get the mean of the logarithms of the 30 p-values, but I am not sure how much it makes sense. How can I solve this problem?

$\endgroup$
5
  • $\begingroup$ You get 0 for what exactly? - the test statistic or its p-value? Can you explain your calculation? $\endgroup$ Aug 16 '15 at 7:10
  • $\begingroup$ I get 0 for the p-value of the statistic. I am doing the calculation using pchisq(-2*sum(log(pvals)), 2*length(pvals), lower.tail=FALSE) in R. I think what I should do is to add log.p=TRUE to the parameters of pchisq function. $\endgroup$
    – user5054
    Aug 16 '15 at 7:22
  • 1
    $\begingroup$ Edit the question to include that info. - it comes down to evaluating the cdf of the chi-squared distribution far out in the upper tail. $\endgroup$ Aug 16 '15 at 7:29
  • 1
    $\begingroup$ Just curiosity: did R really give you such small p-values?! Most commands only go to 2*10^-16 as far as I know $\endgroup$ Aug 17 '15 at 7:51
  • $\begingroup$ Yes, I used Wilcoxon test (wilcox.test in R) and it gave those p-values. $\endgroup$
    – user5054
    Aug 18 '15 at 3:12
7
$\begingroup$
  1. If underflow is the issue, you could simply compare the sum of their logs (the mean of the logs is entirely equivalent)

    Note that Fishers method actually compares $-2\sum_{i=1}^k \log(p_i)$ to a $\chi^2_{2k}$ ... so Fisher would also be working on the log scale.

  2. However, it's not clear to me that there's necessarily any particularly meaningful comparison between two sets of extremely small p-values. For starters, the values in the extreme tail will tend to be quite sensitive to even small deviations from assumptions.

  3. Some people would argue that you shouldn't compare p-values at all.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.