Alternatives for chi-squared test for independence for tables more than 2 x 2

Question

What are some alternatives to the chi-squared test for categorical variables with tables larger than 2 x 2 and cells with a count less than 5, if I don't want to merge classes?

Answer 1

There are some common misunderstandings here. The chi-squared test is perfectly fine to use with tables that are larger than $2\!\times\! 2$. In order for the actual distribution of the chi-squared test statistic to approximate the chi-squared distribution, the traditional recommendation is that all cells have expected values $\ge 5$. Two things must be noted here:

1. It does not matter what the observed cell counts are—they could well be $0$ with no problem—only the expected counts matter.

2. This traditional rule of thumb is now known to be too conservative. It can be fine to have $\le 20\%$ of the cells with expected counts $< 5$ as long as no expected counts are $<1$. See:

• Campbell Ian, 2007, Chi-squared and Fisher-Irwin tests of two-by-two tables with small sample recommendations, Statistics in Medicine, 26, 3661 - 3675

If your expected counts do not match this more accurate criterion, there are some alternative options available:

1. Your best bet is probably to simulate the sampling distribution of the test statistic, or to use a permutation test. (Note, however, that R's, chisq.test(..., simulate.p.value=TRUE) is really a simulation of Fisher's exact test—see #2—so you'd have to code the simulation manually if you didn't want that.)

2. You could use an alternative test, such as Fisher's exact test. Although Fisher's exact test is often recommended in this situation, it is worth noting that it makes different assumptions and may not be appropriate. Namely, Fisher's exact test assumes the row and column counts were set in advance and only the arrangement of the row x column combinations can vary (see: Given the power of computers these days, is there ever a reason to do a chi-squared test rather than Fisher's exact test?). If you are uncomfortable with this assumption, simulating the chi-squared will be a better option.

Answer 2

As an addition to the answer by @gung, let us look at another way: Simulating the null distribution of the $\chi^2$ test statistic, unconditionally, and not as with the Fisher exact test, use the conditional distribution (conditional on the two margins).

As an example, I will use the $2\times 3$ table from Fisher's exact test in 3x2 contingency table below:

 mydat
  a  b c
A 2 12 1
B 5  3 1

We must calculate (really estimate) the marginal probabilities from this table, and then use this to estimate the probabilities of the table, under independence. Then we use this in multinomial sampling, with the same sample size.

One problem which shows, is that some (in this example more than 10%) of the simulated tables have some zero margins. That makes for a problem in calculating chi-square, since it means that we are dividing by zero. My R functions return then NaN (Not a Number). For the illustrations below, I have decided to simply remove this values, although that is certainly debatable.

I get the following simulated null distribution:

overlaid in red is the density of the usual approximate chi-square distribution with two degrees of freedom.

We can use this to calculate a simulated p-value, which turns out as 0.0523 as compared to the usual p-value, as returned by R's chisq.test, which is 0.06293. The simulation option of that function gives a p-value of 0.04927.

But, the problem of what to do in the case of simulated tables with null margins is interesting. It shows that simulating the null distribution of the chi-squared test statistic, unconditionally, is not unproblematic. The results above is not really unconditional, they are conditional on no null margins. Using the Fisher exact test, pragmatically avoids the issue!