97
$\begingroup$

Given that software can do the Fisher's exact test calculation so easily nowadays, is there any circumstance where, theoretically or practically, the chi-squared test is actually preferable to Fisher's exact test?

Advantages of the Fisher's exact test include:

  • scaling to contingency tables larger than 2x2 (i.e any r x c table)
  • gives an exact p-value
  • not needing to have a minimum expected cell count to be valid
$\endgroup$
9
  • 12
    $\begingroup$ Because it is good old classics. Soon it will become exquisite vintage. Thereafter yet, when people rise against computers it shall live its second youth. $\endgroup$
    – ttnphns
    Commented Aug 13, 2011 at 20:12
  • 9
    $\begingroup$ Have you ever tried to compute Fisher's exact test statistic on a large table? (It takes too long...) $\endgroup$
    – whuber
    Commented Aug 13, 2011 at 20:44
  • 25
    $\begingroup$ In addition to the good comments and answers you've already gotten, I think the better question is "Given the power of computers, why not do simulation/permutation tests all the time?". $\endgroup$
    – Peter Flom
    Commented Aug 14, 2011 at 12:44
  • 1
    $\begingroup$ @whuber I did a (proprietary) implementation without (large numbers of) tables, in C++. It runs thousands of P values for numbers up to 8 digits in seconds. $\endgroup$ Commented Sep 8, 2011 at 19:31
  • 1
    $\begingroup$ @Michel I meant the total number of cells in the table. The calculation is easy for 2 x 2 tables, but as the tables grow large, the calculations become onerous. $\endgroup$
    – whuber
    Commented Sep 12, 2011 at 22:03

2 Answers 2

77
$\begingroup$

You can turn the question around. Since the ordinary Pearson $\chi^2$ test is almost always more accurate than Fisher's exact test and is much quicker to compute, why does anyone use Fisher's test?

Note that it is a fallacy that the expected cell frequencies have to exceed 5 for Pearson's $\chi^2$ to yield accurate $P$-values. The test is accurate as long as expected cell frequencies exceed 1.0 if a very simple $\frac{N-1}{N}$ correction is applied to the test statistic.


From R-help, 2009:

Campbell, I. Chi-squared and Fisher-Irwin tests of two-by-two tables with small sample recommendations. Statistics in Medicine 2007; 26:3661-3675. (abstract)

  • ...latest edition of Armitage's book recommends that continuity adjustments never be used for contingency table chi-square tests;

  • E. Pearson modification of Pearson chi-square test, differing from the original by a factor of (N-1)/N;

  • Cochran noted that the number 5 in "expected frequency less than 5" was arbitrary;

  • findings of published studies may be summarized as follows, for comparative trials:

  1. Yates' chi-squared test has type I error rates less than the nominal, often less than half the nominal;

  2. The Fisher-Irwin test has type I error rates less than the nominal;

  3. K Pearson's version of the chi-squared test has type I error rates closer to the nominal than Yate's chi-squared test and the Fisher-Irwin test, but in some situations gives type I errors appreciably larger than the nominal value;

  4. The 'N-1' chi-squared test, behaves like K. Pearson's 'N' version, but the tendency for higher than nominal values is reduced;

  5. The two-sided Fisher-Irwin test using Irwin's rule is less conservative than the method doubling the one-sided probability;

  6. The mid-P Fisher-Irwin test by doubling the one-sided probability performs better than standard versions of the Fisher-Irwin test, and the mid-P method by Irwin's rule performs better still in having actual type I errors closer to nominal levels.";

  • strong support for the 'N-1' test provided expected frequencies exceed 1;

  • flaw in Fisher test which was based on Fisher's premise that marginal totals carry no useful information;

  • demonstration of their useful information in very small sample sizes;

  • Yates' continuity adjustment of N/2 is a large over-correction and is inappropriate;

  • counter arguments exist to the use of randomization tests in randomized trials;

  • calculations of worst cases;

  • overall recommendation: use the 'N-1' chi-square test when all expected frequencies are at least 1; otherwise use the Fisher-Irwin test using Irwin's rule for two-sided tests, taking tables from either tail as likely, or less, as that observed; see letter to the editor by Antonio Andres and author's reply in 27:1791-1796; 2008.


Crans GG, Shuster JJ. How conservative is Fisher's exact test? A quantitative evaluation of the two-sample comparative binomial trial. Statistics in Medicine 2008; 27:3598-3611. (abstract)

  • ...first paper to truly quantify the conservativeness of Fisher's test;

  • "the test size of FET was less than 0.035 for nearly all sample sizes before 50 and did not approach 0.05 even for sample sizes over 100.";

  • conservativeness of "exact" methods;

  • see Stat in Med 28:173-179, 2009 for a criticism which was unanswered


Lydersen S, Fagerland MW, Laake P. Recommended tests for association in $2\times 2$ tables. Statistics in Medicine 2009; 28:1159-1175. (abstract)

  • ...Fisher's exact test should never be used unless the mid-$P$ correction is applied;

  • value of unconditional tests;

  • see letter to the editor 30:890-891;2011

$\endgroup$
7
  • 2
    $\begingroup$ Can you suggest how to apply the (N-1)/N correction? Are there any online calculators that incorporate this correction? Is there an easy way to manually adjust the results of the chi-squared test to make this correction yourself? $\endgroup$
    – D.W.
    Commented Aug 15, 2011 at 4:16
  • $\begingroup$ One of the references I listed above is your best bet. $\endgroup$ Commented Aug 15, 2011 at 5:43
  • 2
    $\begingroup$ Why are you saying the $\chi^2$ "is almost always more accurate than Fisher's exact test" ? I would say the converse, because the $\chi^2$ is not an "exact" test. $\endgroup$ Commented Aug 7, 2015 at 12:46
  • 4
    $\begingroup$ Labeling something as "exact" does not make it so. See the wonderful explanation below by @suncoolsu that you must have missed (you also missed all the explanations above). The Pearson test is even more accurate than Pearson thought it was. See citeulike.org/user/harrelfe/article/13265687 and citeulike.org/user/harrelfe/article/13263676 for example. Fisher's "exact" test is only exact in the sense that the true type I error is not larger than claimed. But it turns out to be smaller than claimed, so the type II error is higher, meaning less power. $\endgroup$ Commented Aug 7, 2015 at 12:57
  • 1
    $\begingroup$ In case anyone ever searches for a neat implementation of this great post and reference to the campell (2007) paper, you can find it here: stats.stackexchange.com/a/439226/250598 $\endgroup$
    – Björn
    Commented Jul 22, 2022 at 23:21
51
$\begingroup$

This is a great question.

Fisher's exact test is one of the great examples of Fisher's clever use of experimental design, together with conditioning on data (basically on tables with the observed row and marginal totals) and his ingenuity at finding probability distributions (though this isn't the best example, for a better example see here). The use of computers to calculate "exact" p-values has definitely helped to obtain accurate answers.

However, it is hard to justify the assumptions of Fisher's exact test in practice. Because the so called "exact" comes from the fact that in the "tea tasting experiement" or in the 2x2 contingency tables case, the row total and column total, that is, the marginal totals are fixed by design. This assumption is rarely justified in practice. For nice references see here.

The name "exact" leads one into believing that the p-values given by this test are exact, which again in most of the cases is unfortunately not correct because of these reasons

  1. If the marginals are not fixed by design (which happens almost every time in practice), the p-values will be conservative.
  2. Since the test uses a discrete probability distribution (specifically, Hyper-geometric distribution), for certain cutoffs it is impossible to calculate the "exact null probabilities", that is, p-value.

In most of the practical cases, using a likelihood ratio test or Chi-square test should not give very different answers (p-value) from a Fisher's exact test. Yes, when the marginals are fixed, Fisher's exact test is a better choice, but this will happen rarely. Therefore, using Chi-square test of likelihood ratio test is always recommended for consistency checks.

Similar ideas apply when the Fisher's exact test is generalized to any table, which basically equivalent to calculating Multivariate Hypergeometric proabilities. Therefore one must always try to calculate Chi-square and likelihood ratio distribution based p-values, in addition to "exact" p-values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.