Fisher exact test is said to be used with a total sample (n) < 1000, whereas chi-squared test should be used when each category (/cell in a contingency table) >=5. What if you have an mxn contingency table where the total sample size > 1000 but some of the cells have 0 or 1 sample size?

  • 1
    $\begingroup$ The rule of thumb $\ge 5$ applies to expected frequencies. It's widely considered very conservative, and I could cite authorities for $\ge 1$ as a fairly safe rule of thumb. But it's always good advice to watch out when you have very small expected frequencies. For 0 or 1 "sample size" here, read "observed frequency"? $\endgroup$ – Nick Cox May 20 '16 at 16:26
  • $\begingroup$ Correct Nick; sample sizes here I was always referring to observed frequencies. $\endgroup$ – dter May 20 '16 at 16:34
  • $\begingroup$ I'd take a step back and consider whether a hypothesis of no association between variables makes sense any way. When some categories in a large sample are very rare, it's often a sign that you are a long way from independence of rows and columns. That's certainly not a rule as small frequencies can be expected under independence, but I've often seen it in practice. $\endgroup$ – Nick Cox May 20 '16 at 16:37

How far above 1,000 is your sample size? If it's not far above 1,000, you can use Fisher's exact test - it's simply recommended that you don't because of computational limitations.

If Fisher's exact test is too computationally intensive and you need the chi-square test, would try to "bin" the variables differently. That is, collapse categories until you have at least 5 in each cell. You could, alternatively, use Yates' correction to account for the undercounts in certain cells.

  • $\begingroup$ Sample size most of the time is around 1000 but can go to 1,500 (or more). What makes things 'worse' is the fact that even after computing fisher's exact test I would need to do all pair-wise comparisons (and FDR correction) in order to determine which pair was significant and to calculate the order ratio or another measure of effect size. With chi-squared this is very convenient for mxn matrix with the Cramer-V and residuals to determine which group is 'significant'; no need for pair-wise comparisons. $\endgroup$ – dter May 20 '16 at 16:38
  • $\begingroup$ I've not seen Yates' correction used in decades. Difficult to know how far that is fashion and how far logic. $\endgroup$ – Nick Cox May 20 '16 at 16:38
  • $\begingroup$ Let's assume the expected frequencies are all >=5 (although from what I've been reading this is not really 'true' link ), if the total sample size is say 50, would a chi square perform 'poorly' compared to a fisher? (since Fisher is said to be better for smaller sample sizes). $\endgroup$ – dter May 20 '16 at 16:48
  • $\begingroup$ For a one-off problem when assumptions are in doubt I see no great difficulty in trying both Fisher and chi-square tests. If they both agree, there's some reassurance; if they disagree you at least know to tread carefully. $\endgroup$ – Nick Cox May 21 '16 at 10:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.