Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Given that software can do the Fisher's exact test calculation so easily nowadays, is there any circumstance where, theoretically or practically, the chi-squared test is actually preferable to Fisher's exact test?

Advantages of the Fisher's exact test include:

  • scaling to contingency tables larger than 2x2 (i.e any r x c table)
  • gives an exact p-value
  • not needing to have a minimum expected cell count to be valid

Thanks for any opinions!

share|improve this question
7  
Because it is good old classics. Soon it will become exquisite vintage. Thereafter yet, when people rise against computers it shall live its second youth. –  ttnphns Aug 13 '11 at 20:12
6  
Have you ever tried to compute Fisher's exact test statistic on a large table? (It takes too long...) –  whuber Aug 13 '11 at 20:44
14  
In addition to the good comments and answers you've already gotten, I think the better question is "Given the power of computers, why not do simulation/permutation tests all the time?". –  Peter Flom Aug 14 '11 at 12:44
    
This would be better put as a comment to the question than as an answer –  probabilityislogic Aug 14 '11 at 13:12
1  
@Michel I meant the total number of cells in the table. The calculation is easy for 2 x 2 tables, but as the tables grow large, the calculations become onerous. –  whuber Sep 12 '11 at 22:03

2 Answers 2

You can turn the question around. Since the ordinary Pearson $\chi^2$ test is almost always more accurate than Fisher's exact test and is much quicker to compute, why does anyone use Fisher's test?

Note that it is a fallacy that the expected cell frequencies have to exceed 5 for Pearson's $\chi^2$ to yield accurate $P$-values. The test is accurate as long as expected cell frequencies exceed 1.0 if a very simple $\frac{N-1}{N}$ correction is applied to the test statistic.


http://onlinelibrary.wiley.com/doi/10.1002/sim.2832/abstract:

From R-help, 2009:

@Article{cam07chi,
author = {Campbell, Ian}, title = {Chi-squared and {Fisher-Irwin} tests of two-by-two tables with small sample recommendations}, journal = Stat in Med, year = 2007, volume = 26, pages = {3661-3675}, annote = {2x2 table;chi-squared test;Fisher-Irwin test;exact tests;small sample recommendations;

  • ...latest edition of Armitage's book recommends that continuity adjustments never be used for contingency table chi-square tests;

  • E. Pearson modification of Pearson chi-square test, differing from the original by a factor of (N-1)/N;

  • Cochran noted that the number 5 in "expected frequency less than 5" was arbitrary;

  • findings of published studies may be summarized as follows, for comparative trials:

    1. Yate's chi-squared test has type I error rates less than the nominal, often less than half the nominal;

    2. The Fisher-Irwin test has type I error rates less than the nominal;

    3. K Pearson's version of the chi-squared test has type I error rates closer to the nominal than Yate's chi-squared test and the Fisher-Irwin test, but in some situations gives type I errors appreciably larger than the nominal value;

    4. The 'N-1' chi-squared test, behaves like K. Pearson's 'N' version, but the tendency for higher than nominal values is reduced;

    5. The two-sided Fisher-Irwin test using Irwin's rule is less conservative than the method doubling the one-sided probability;

    6. The mid-P Fisher-Irwin test by doubling the one-sided probability performs better than standard versions of the Fisher-Irwin test, and the mid-P method by Irwin's rule performs better still in having actual type I errors closer to nominal levels.";

  • strong support for the 'N-1' test provided expected frequencies exceed 1;

  • flaw in Fisher test which was based on Fisher's premise that marginal totals carry no useful information;

  • demonstration of their useful information in very small sample sizes;

  • Yate's continuity adjustment of N/2 is a large over correction and is inappropriate;

  • counter arguments exist to the use of randomization tests in randomized trials;

  • calculations of worst cases;

  • overall recommendation: use the 'N-1' chi-square test when all expected frequencies are at least 1, otherwise use the Fisher-Irwin test using Irwin's rule for two-sided tests, taking tables from either tail as likely, or less, as that observed; see letter to the editor by Antonio Andres and author's reply in 27:1791-1796; 2008.


http://onlinelibrary.wiley.com/doi/10.1002/sim.3221/abstract:

@Article{cra08how, author = {Crans, Gerald G. and Shuster, Jonathan J.}, title = {How conservative is {Fisher's} exact test? {A} quantitative evaluation of the two-sample comparative binomial trial}, journal = Stat in Med, year = 2008, volume = 27, pages = {3598-3611}, annote = {Fisher's exact test; $2\times 2$ contingency table;size of test; comparative binomial experiment;

  • ...first paper to truly quantify the conservativeness of Fisher's test;

  • "the test size of FET was less than 0.035 for nearly all sample sizes before 50 and did not approach 0.05 even for sample sizes over 100.";

  • conservativeness of "exact" methods;

  • see Stat in Med 28:173-179, 2009 for a criticism which was unanswered


http://onlinelibrary.wiley.com/doi/10.1002/sim.3531/abstract:

@Article{lyd09rec, author = {Lydersen, Stian and Fagerland, Morten W. and Laake, Petter}, title = {Recommended tests for association in $2\times 2$ tables}, journal = Stat in Med, year = 2009, volume = 28, pages = {1159-1175}, annote = {$2\times 2$ tables;

  • ...Fisher's exact test should never be used unless the mid-$P$ correction is applied;

  • value of unconditional tests;

  • see letter to the editor 30:890-891;2011

share|improve this answer
    
Can you suggest how to apply the (N-1)/N correction? Are there any online calculators that incorporate this correction? Is there an easy way to manually adjust the results of the chi-squared test to make this correction yourself? –  D.W. Aug 15 '11 at 4:16
    
One of the references I listed above is your best bet. –  Frank Harrell Aug 15 '11 at 5:43
3  
Thanks @whuber for the great edits. –  Frank Harrell Aug 16 '11 at 15:49

This is a great question.

Fisher's exact test is one of the great examples of Fisher's clever use of experimental design, together with conditioning on data (basically on tables with the observed row and marginal totals) and his ingenuity at finding probability distributions (though this isn't the best example, for a better example see here). The use of computers to calculate "exact" p-values has definitely helped to obtain accurate answers.

However, it is hard to justify the assumptions of Fisher's exact test in practice. Because the so called "exact" comes from the fact that in the "tea tasting experiement" or in the 2x2 contingency tables case, the row total and column total, that is, the marginal totals are fixed by design. This assumption is rarely justified in practice. For nice references see here.

The name "exact" leads one into believing that the p-values given by this test are exact, which again in most of the cases is unfortunately not correct because of these reasons

  1. If the marginals are not fixed by design (which happens almost every time in practice), the p-values will be conservative.
  2. Since the test uses a discrete probability distribution (specifically, Hyper-geometric distribution), for certain cutoffs it is impossible to calculate the "exact null probabilities", that is, p-value.

In most of the practical cases, using a likelihood ratio test or Chi-square test should not give very different answers (p-value) from a Fisher's exact test. Yes, when the marginals are fixed, Fisher's exact test is a better choice, but this will happen rarely. Therefore, using Chi-square test of likelihood ratio test is always recommended for consistency checks.

Similar ideas apply when the Fisher's exact test is generalized to any table, which basically equivalent to calculating Multivariate Hypergeometric proabilities. Therefore one must always try to calculate Chi-square and likelihood ratio distribution based p-values, in addition to "exact" p-values.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.