3
$\begingroup$

I'm comparing success rates for two repeated experiments with $n_1$ and $n_2$ successes out of $N$ trials. $N$ is in the order of $10^7$. I don't know beforehand which experiment has a lower success rate, so I'd probably use a two-sided test. I see two options:

  1. The test of equal proportions (in R: prop.test(c(n1, n2), c(N, N)))
  2. Fisher's exact test (in R: fisher.test(matrix(c(n1, n2, N-n1, N-n2), ncol = 2)))

Now, for $N$ that large, Fisher's exact test is slow. (The implementation seems to evaluate the density of the hypergeometric distribution on a support of the order of $N$.) However, the test of equal proportions seems to have less power.

Does the test of equal proportions always return a p-value not less than that of Fisher's exact test for the same data? Is there a more powerful alternative to the test of equal proportions that isn't that expensive computation-wise?

EDIT: A computational test on 1000 matrices with "almost equal" entries suggests that the p-value computed by Fisher's test is almost always, but not always, larger. I'm still looking for stronger argumentation, and a faster/more powerful test.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
+50
$\begingroup$

prop.test uses a Pearson chi-square test. This is an asymptotic test. It will be worst when you have small samples or get too near the tails. Fishers will always be "better" because it is an "exact" test that does not rely upon asymptotic arguments to obtain its p-values...rather, it computes all the ways the table could have come about and then finds the proportion that were as-or-more-extreme.

Practically, this will result in Fisher's being less "powerful" when it matters because Pearson's approximation is most wrong in exactly those cases.

I do not know why fisher.test should take so long. For sample sizes on the order of $10^7$, it should have dropped to approximate methods unless the events are really rare. Are they? An alternative might be binom.test which uses Fisher's and may swap algorithms when sample sizes get large and event rates are still common. That might speed things up. A MonteCarlo version might work, also.

In your case and for sample sizes this high and non-rare events, Fisher's and Pearson's should not disagree to any real extent but I'd request the continuity-correction on Pearson prop.test(..., correct=TRUE). Try your simulation with this option and see if there is a dime's worth of difference then.

Another option is Barnard's unconditional test which can be more powerful but which many people frown at (even Barnard) though their cited reasons are often esoteric. In any case, that is not likely to be faster than either Pearson or Fisher.

Improve this answer
$\endgroup$
9
  • $\begingroup$ Thanks. I don't understand the second paragraph. fisher.test() always computes exact values and doesn't rely on approximations at all. My question mostly asks about which test leads lower p-values (am I right that such a test will have more power?). -- I have both large samples and (in some cases) rare events. My application might be somewhat unusual -- I'm collecting many p-values from similar experiments and combining them through meta-analysis in order to "prove" the null hypothesis (or at least be reasonably sure about it). That's why I think I need "accurate" p-values. $\endgroup$
    – krlmlr
    Jan 6, 2016 at 19:49
  • $\begingroup$ I do not think Pearson MUST lead to lower p-values than Fisher's ALWAYS (though I'm trying to imagine how the discrete jaggies are enveloped by the chi-square). I'm more certain that a Yates-corrected Pearson will not ALWAYS lead to lower p-values than Fisher's. When they basically agree, it does not matter. When they do not agree, Fisher will be the one raining on your parade! Really, though, try adding the continuity-correction to prop.test. See if the sun doesn't come back out. $\endgroup$
    – StatNoodle
    Jan 6, 2016 at 20:13
  • $\begingroup$ Oh, one other comment. Remember, Pearson is really requires a minimum of 5 events (and non-events) to be legitimate. Beware the tails...there be dragons there. If you have cases rarer than that, you'll need Fisher. $\endgroup$
    – StatNoodle
    Jan 6, 2016 at 20:15
  • $\begingroup$ And one more comment. Fisher can be implemented with approximations for large sample and large number of category cases. I do not know how fisher.test is implemented. But generally, if it is taking too long, it is because the sample sizes are large or the number of categories is large. Large sample sizes can be handled by approximating the distributions. See the Wikipedia entry on Fisher's Exact Test for details. But I did not write fisher.test() nor do I know what they did. Test from the exact package might do better. $\endgroup$
    – StatNoodle
    Jan 6, 2016 at 20:18
  • $\begingroup$ The hint on the continuity correction was gold, thanks! When I re-run my script, the p-value computed by Fisher's test is now always larger than that computed by chi-square. Does this mean that the chi-square test without continuity correction is more powerful? What are the drawbacks, especially when I'm interested in checking uniformity of the distribution of an array of p-values? $\endgroup$
    – krlmlr
    Jan 6, 2016 at 21:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.