15
$\begingroup$

What are some alternatives to the chi-squared test for categorical variables with tables larger than 2 x 2 and cells with a count less than 5, if I don't want to merge classes?

$\endgroup$
3
  • 5
    $\begingroup$ The Chi-Square-test can also be used with larger tables than 2x2. Could you explain why the Chi-Square-test should not be appropriate for your problem? Additionally, could you state the problem you're hoping to solve? $\endgroup$ Jun 28, 2015 at 16:33
  • $\begingroup$ I have a 2 x 3 contingency table, and cells with a count less than 5 $\endgroup$
    – Israel
    Jun 28, 2015 at 16:37
  • 4
    $\begingroup$ Thanks, please edit your question and add this information as not everyone reads the comments. A usual rule of thumb regarding the Chi-Square-test is that its results can be inaccurate if the expected cell counts are lower than 5. Usually, a Fisher-Test is recommended in these cases. Barnard's test may also be an option. $\endgroup$ Jun 28, 2015 at 16:52

2 Answers 2

25
$\begingroup$

There are some common misunderstandings here. The chi-squared test is perfectly fine to use with tables that are larger than $2\!\times\! 2$. In order for the actual distribution of the chi-squared test statistic to approximate the chi-squared distribution, the traditional recommendation is that all cells have expected values $\ge 5$. Two things must be noted here:

  1. It does not matter what the observed cell counts are—they could well be $0$ with no problem—only the expected counts matter.

  2. This traditional rule of thumb is now known to be too conservative. It can be fine to have $\le 20\%$ of the cells with expected counts $< 5$ as long as no expected counts are $<1$. See:

If your expected counts do not match this more accurate criterion, there are some alternative options available:

  1. Your best bet is probably to simulate the sampling distribution of the test statistic, or to use a permutation test. (Note, however, that R's, chisq.test(..., simulate.p.value=TRUE) is really a simulation of Fisher's exact test—see #2—so you'd have to code the simulation manually if you didn't want that.)

  2. You could use an alternative test, such as Fisher's exact test. Although Fisher's exact test is often recommended in this situation, it is worth noting that it makes different assumptions and may not be appropriate. Namely, Fisher's exact test assumes the row and column counts were set in advance and only the arrangement of the row x column combinations can vary (see: Given the power of computers these days, is there ever a reason to do a chi-squared test rather than Fisher's exact test?). If you are uncomfortable with this assumption, simulating the chi-squared will be a better option.

$\endgroup$
5
  • 2
    $\begingroup$ In the help manual of the function chisq.test() it says 'simulation is done by random sampling from the set of all contingency tables with given marginals [...] Note that this is not the usual sampling situation assumed for the chi-squared test but rather that for Fisher's exact test.'. So if we set simulate.p.values=TRUE it also assumes fixed margins (just like the Fisher's exact test). $\endgroup$
    – retodomax
    Mar 29, 2022 at 13:18
  • 1
    $\begingroup$ @retodomax, you're right about that. I'll update the answer. $\endgroup$ Mar 29, 2022 at 15:20
  • 2
    $\begingroup$ Fisher's exact test assumes the row and column counts were set in advance ... This is not really true, it conditions on the observed margins. see stats.stackexchange.com/questions/441139/… $\endgroup$ Mar 29, 2022 at 15:44
  • 1
    $\begingroup$ It's interesting he says that, @kjetilbhalvorsen, but lots of people in published statistical literature discuss Fisher's exact test this way, eg, Gelman (p 375, pdf). $\endgroup$ Mar 29, 2022 at 18:50
  • $\begingroup$ @gung: I have seen before Gelman saying that. As a bayesian, maybe he does not believe in frequentist conditioning. The argument is that the marginals are approximate ancillary (there is a paper by Barnard showing they are not exactly ancillary). But there cannot be much info in the marginals only about dependence or independence. So the logic is as with other frequentist conditional tests. $\endgroup$ Mar 29, 2022 at 20:26
2
$\begingroup$

As an addition to the answer by @gung, let us look at another way: Simulating the null distribution of the $\chi^2$ test statistic, unconditionally, and not as with the Fisher exact test, use the conditional distribution (conditional on the two margins).

As an example, I will use the $2\times 3$ table from Fisher's exact test in 3x2 contingency table below:

 mydat
  a  b c
A 2 12 1
B 5  3 1

We must calculate (really estimate) the marginal probabilities from this table, and then use this to estimate the probabilities of the table, under independence. Then we use this in multinomial sampling, with the same sample size.

One problem which shows, is that some (in this example more than 10%) of the simulated tables have some zero margins. That makes for a problem in calculating chi-square, since it means that we are dividing by zero. My R functions return then NaN (Not a Number). For the illustrations below, I have decided to simply remove this values, although that is certainly debatable.

I get the following simulated null distribution:

Simulated null distribution of chi-square

overlaid in red is the density of the usual approximate chi-square distribution with two degrees of freedom.

We can use this to calculate a simulated p-value, which turns out as 0.0523 as compared to the usual p-value, as returned by R's chisq.test, which is 0.06293. The simulation option of that function gives a p-value of 0.04927.

But, the problem of what to do in the case of simulated tables with null margins is interesting. It shows that simulating the null distribution of the chi-squared test statistic, unconditionally, is not unproblematic. The results above is not really unconditional, they are conditional on no null margins. Using the Fisher exact test, pragmatically avoids the issue!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.