Suppose I have two populations, A and B, observed counts of whether population members are in one of two classes:

         popA  popB
class 0   100   1
class 1   200   99

I'd like to have an measure of how likely it is that these populations are different. The Fisher Exact Test seems like one good answer. However, its horrendously slow because of its factorial calculation (millions of tests will take ~1 day for me, using scipy.stats).

Are there faster approximations that are available?

Mostly I'm interested in using this as a splitting criteria for decision trees (popB will be a subset of popA and I'd like test how dissimilar a sub-population B is from the original population A). Common spitting criteria (gini, entropy ratio) don't take the amount of evidence in both populations into account.

  • $\begingroup$ The usual hypothesis tests don't address "how likely it is that these populations are different". If that's really what you seek, Fisher's exact test is no use at all. I presume this stems from a misunderstanding of what a p-value is. With large marginal totals, the usual chi-squared test should be sufficient, depending on what you actually need. However, a smart implementation of Fisher's exact test should be able to do this case in no time. In fact, I think I might be able to do this one with a pen and paper. $\endgroup$
    – Glen_b
    Sep 22, 2016 at 5:25
  • 1
    $\begingroup$ However in cases that really would take a long time you can always simulate from random tables with the same marginals using Patefield's algorithm and get a randomization-test approximation to the p-value. $\endgroup$
    – Glen_b
    Sep 22, 2016 at 5:33
  • 1
    $\begingroup$ Well, it's a little tedious to do by hand (because the other tail in a two tailed test has a bunch of more extreme cases) but there's only about fifty hypergeometric probabilities to calculate; it should take you a fraction of a second. $\endgroup$
    – Glen_b
    Sep 22, 2016 at 5:46
  • 1
    $\begingroup$ Here's one calculation of it done in R (not even the most efficient way, but it should be obvious what's being calculated): sum(dhyper(c(0:1,55:100),101,299,100)) ... which gives [1] 1.092822e-13 ... which is the same answer as you get with fisher.test(matrix(c(1,99,100,200),nr=2)) $\endgroup$
    – Glen_b
    Sep 22, 2016 at 6:49
  • $\begingroup$ The sampling methods are similarly expense when making millions of tests. Just tried Chi2 and it seems to work great. $\endgroup$
    – user48956
    Sep 22, 2016 at 18:50


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.