Fisher exact test is said to be used with a total sample (n) < 1000, whereas chi-squared test should be used when each category (/cell in a contingency table) >=5. What if you have an mxn contingency table where the total sample size > 1000 but some of the cells have 0 or 1 sample size?
3
-
1$\begingroup$ The rule of thumb $\ge 5$ applies to expected frequencies. It's widely considered very conservative, and I could cite authorities for $\ge 1$ as a fairly safe rule of thumb. But it's always good advice to watch out when you have very small expected frequencies. For 0 or 1 "sample size" here, read "observed frequency"? $\endgroup$– Nick CoxCommented May 20, 2016 at 16:26
-
$\begingroup$ Correct Nick; sample sizes here I was always referring to observed frequencies. $\endgroup$– dterCommented May 20, 2016 at 16:34
-
$\begingroup$ I'd take a step back and consider whether a hypothesis of no association between variables makes sense any way. When some categories in a large sample are very rare, it's often a sign that you are a long way from independence of rows and columns. That's certainly not a rule as small frequencies can be expected under independence, but I've often seen it in practice. $\endgroup$– Nick CoxCommented May 20, 2016 at 16:37
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
How far above 1,000 is your sample size? If it's not far above 1,000, you can use Fisher's exact test - it's simply recommended that you don't because of computational limitations.
If Fisher's exact test is too computationally intensive and you need the chi-square test, would try to "bin" the variables differently. That is, collapse categories until you have at least 5 in each cell. You could, alternatively, use Yates' correction to account for the undercounts in certain cells.
answered May 20, 2016 at 16:21
-
$\begingroup$ Sample size most of the time is around 1000 but can go to 1,500 (or more). What makes things 'worse' is the fact that even after computing fisher's exact test I would need to do all pair-wise comparisons (and FDR correction) in order to determine which pair was significant and to calculate the order ratio or another measure of effect size. With chi-squared this is very convenient for mxn matrix with the Cramer-V and residuals to determine which group is 'significant'; no need for pair-wise comparisons. $\endgroup$– dterCommented May 20, 2016 at 16:38
-
$\begingroup$ I've not seen Yates' correction used in decades. Difficult to know how far that is fashion and how far logic. $\endgroup$– Nick CoxCommented May 20, 2016 at 16:38
-
$\begingroup$ Let's assume the expected frequencies are all >=5 (although from what I've been reading this is not really 'true' link ), if the total sample size is say 50, would a chi square perform 'poorly' compared to a fisher? (since Fisher is said to be better for smaller sample sizes). $\endgroup$– dterCommented May 20, 2016 at 16:48
-
$\begingroup$ For a one-off problem when assumptions are in doubt I see no great difficulty in trying both Fisher and chi-square tests. If they both agree, there's some reassurance; if they disagree you at least know to tread carefully. $\endgroup$– Nick CoxCommented May 21, 2016 at 10:03