5
$\begingroup$

I have a bunch of independent p-values and now I want to combine them using the Fisher's method. Each of the individual p-values is coming from a one-sided test. I am just a little bit confused about the "side" of the Fisher's method test, i.e. when I calculate the Fisher's method p-value in R, I use:

1 - pchisq( -2*sum(log(p-values)), df)

where df = 2*length(p-values).

Is this a one-sided test? It should be because when the test statistic -2*sum(log(p-values)) is much smaller than df, then the Fisher's p-value is close to 1. There should be a problem here, right? How shall I (or should I?) reject the null if my test statistic is very small? I am just uncomfortable with the "close-to-1" p-values.

By the way, I use the method for testing model goodness-of-fit, and a small p-value is to indicate model lack-of-fit.

$\endgroup$
9
$\begingroup$

i) First, a recommendation:

Use pchisq( -2*sum(log(p-values)), df, lower.tail=FALSE) instead of 1- ... - you're likely to end up with more accuracy for small p-values. To see that they're sometimes going to give different results, try this:

 x=70;c(1-pchisq(x,1),pchisq(x,1,lower.tail=FALSE))

ii) Yes, it's one-sided. Small values of the chi-square statistic indicate that the component p-values tend to be large (that is, a lack of evidence against the overall null). Imagine you were doing a t-test and the sample means were really, really close together... i.e. $|t|$ was unusually small. Would you reject the null hypothesis that they were equal because they were unusually close together?

Clearly not. You might conclude something else was wrong (like one of your assumptions could be faulty, or you used a really bad test, or your calculation might be wrong, or someone fiddled the data, or ...) - but you wouldn't conclude the means were different because they were surprisingly close!

Indeed - what would you do in that situation:

> t.test(x,y,var.equal=TRUE)

    Two Sample t-test

data:  x and y
t = 1e-04, df = 18, p-value = 0.9999
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.7213824  0.7214315
sample estimates:
 mean of x  mean of y 
-0.2161466 -0.2161711 

So there's a two sample t-test with $p$ really close to 1 (~0.999944). What do you conclude?

So now, with a goodness of fit, what kinds of things might a p-value really close to 1 tell you?

$\endgroup$
  • $\begingroup$ thank you so much for the answer! For a two-sample t-test with p -> 1, I conclude no evidence of a difference between the two samples. For a GOF test p -> 1, I conclude there might be some "super-fitting" issue? My test works fine for other situations, but only when testing a certain dispersion model will it give me such a large p-value. Thanks again! $\endgroup$ – alittleboy May 18 '13 at 16:07
  • $\begingroup$ oh, one more thing, the use of 1-pchisq(...) seems to be equivalent to pchisq(..., lower.tail=FALSE) -- they give identical p-values. $\endgroup$ – alittleboy May 18 '13 at 16:13
  • $\begingroup$ Yes, the "problem" is that the fit is surprisingly good. Maybe too good. But without a better understanding of the situation I can't suggest which of about 6 possibilities, maybe more, might cause it, and even with details I mightn't be able to say. $\endgroup$ – Glen_b May 19 '13 at 1:04
  • 1
    $\begingroup$ On the p-value thing - yes, it's supposed to give the same answer, almost always. It's simply a numerical accuracy issue for extremely small p-values, specifically avoiding a subtraction of the form: $1 - a$ where $a=(1-\epsilon)$, which can sometimes result in a very poor approximation of $\epsilon$. This is known as catastrophic cancellation (see the second sentence here). The people who write the code for the extreme upper tail - and I have not checked the C code for this - presumably are already aware of this well-known issue. $\endgroup$ – Glen_b May 19 '13 at 1:15
  • 1
    $\begingroup$ Compare: x=70;c(1-pchisq(x,1),pchisq(x,1,lower.tail=FALSE)) $\endgroup$ – Glen_b May 19 '13 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.