6
$\begingroup$

I have a bunch of independent p-values and now I want to combine them using the Fisher's method. Each of the individual p-values is coming from a one-sided test. I am just a little bit confused about the "side" of the Fisher's method test, i.e. when I calculate the Fisher's method p-value in R, I use:

1 - pchisq( -2*sum(log(p-values)), df)

where df = 2*length(p-values).

Is this a one-sided test? It should be because when the test statistic -2*sum(log(p-values)) is much smaller than df, then the Fisher's p-value is close to 1. There should be a problem here, right? How shall I (or should I?) reject the null if my test statistic is very small? I am just uncomfortable with the "close-to-1" p-values.

By the way, I use the method for testing model goodness-of-fit, and a small p-value is to indicate model lack-of-fit.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
9
$\begingroup$

i) First, a recommendation:

Use pchisq( -2*sum(log(p-values)), df, lower.tail=FALSE) instead of 1- ... - you're likely to end up with more accuracy for small p-values. To see that they're sometimes going to give different results, try this:

 x=70;c(1-pchisq(x,1),pchisq(x,1,lower.tail=FALSE))

ii) Yes, it's one-sided. Small values of the chi-square statistic indicate that the component p-values tend to be large (that is, a lack of evidence against the overall null). Imagine you were doing a t-test and the sample means were really, really close together... i.e. $|t|$ was unusually small. Would you reject the null hypothesis that they were equal because they were unusually close together?

Clearly not. You might conclude something else was wrong (like one of your assumptions could be faulty, or you used a really bad test, or your calculation might be wrong, or someone fiddled the data, or ...) - but you wouldn't conclude the means were different because they were surprisingly close!

Indeed - what would you do in that situation:

> t.test(x,y,var.equal=TRUE)

    Two Sample t-test

data:  x and y
t = 1e-04, df = 18, p-value = 0.9999
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.7213824  0.7214315
sample estimates:
 mean of x  mean of y 
-0.2161466 -0.2161711

So there's a two sample t-test with $p$ really close to 1 (~0.999944). What do you conclude?

So now, with a goodness of fit, what kinds of things might a p-value really close to 1 tell you?

Improve this answer
$\endgroup$
5
  • $\begingroup$ thank you so much for the answer! For a two-sample t-test with p -> 1, I conclude no evidence of a difference between the two samples. For a GOF test p -> 1, I conclude there might be some "super-fitting" issue? My test works fine for other situations, but only when testing a certain dispersion model will it give me such a large p-value. Thanks again! $\endgroup$
    – alittleboy
    Commented May 18, 2013 at 16:07
  • $\begingroup$ oh, one more thing, the use of 1-pchisq(...) seems to be equivalent to pchisq(..., lower.tail=FALSE) -- they give identical p-values. $\endgroup$
    – alittleboy
    Commented May 18, 2013 at 16:13
  • $\begingroup$ Yes, the "problem" is that the fit is surprisingly good. Maybe too good. But without a better understanding of the situation I can't suggest which of about 6 possibilities, maybe more, might cause it, and even with details I mightn't be able to say. $\endgroup$
    – Glen_b
    Commented May 19, 2013 at 1:04
  • 1
    $\begingroup$ On the p-value thing - yes, it's supposed to give the same answer, almost always. It's simply a numerical accuracy issue for extremely small p-values, specifically avoiding a subtraction of the form: $1 - a$ where $a=(1-\epsilon)$, which can sometimes result in a very poor approximation of $\epsilon$. This is known as catastrophic cancellation (see the second sentence here). The people who write the code for the extreme upper tail - and I have not checked the C code for this - presumably are already aware of this well-known issue. $\endgroup$
    – Glen_b
    Commented May 19, 2013 at 1:15
  • 1
    $\begingroup$ Compare: x=70;c(1-pchisq(x,1),pchisq(x,1,lower.tail=FALSE)) $\endgroup$
    – Glen_b
    Commented May 19, 2013 at 1:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.