# Warning in R - Chi-squared approximation may be incorrect

I have data showing fire fighter entrance exam results. I am testing the hypothesis that exam results are dependent on ethnicity. To test this, I ran a Pearson chi-square test in R. The results show what I expected, but it gave a warning that "In chisq.test(a) : Chi-squared approximation may be incorrect."

> a
white black asian hispanic
pass       5     2     2        0
noShow     0     1     0        0
fail       0     2     3        4
> chisq.test(a)

Pearson's Chi-squared test

data:  a
X-squared = 12.6667, df = 6, p-value = 0.04865

Warning message:
In chisq.test(a) : Chi-squared approximation may be incorrect


Does anyone know why it gave a warning? Is it because I am using a wrong method? Thanx in advance.

-

It gave the warning because many of the expected values will be very small and therefore the approximations of p may not be right.

In R you can use chisq.test(a, simulate.p.value = TRUE) to use simulate p values.

However, with such small cell sizes, all estimates will be poor. It might be good to just test pass vs. fail (deleting "no show") either with chi-square or logistic regression. Indeed, since it is pretty clear that the pass/fail grade is a dependent variable, logistic regression might be better.

-
can you please show how can i apply logistic regression ? –  user1883491 Jan 7 at 12:39
I haven't time right now to do so in detail but look at glm and search on logistic regression. –  Peter Flom Jan 7 at 12:49

The issue is that the chi-square approximation to the distribution of the test statistic relies on the counts being roughly normally distributed. If many of the expected counts are very small, the approximation may be poor.

The noshow category will be a big contributor to the problem; one thing to consider is merging noshow and fail. You'll still get the warning but it won't affect the results nearly so much and the distribution should be quite reasonable (the rule that's being applied before the warning is given is too strict).

But in any case, you can deal with the problem very easily in R; set the simulate.p.value argument to TRUE; then you aren't reliant on the chi-square approximation to the distribution of the test statistic.

-

For such small counts, you could use Fisher's exact test:

> fisher.test(a)

Fisher's Exact Test for Count Data

data:  a
p-value = 0.02618
alternative hypothesis: two.sided

-

Please see the "Assumptions" section of Pearson's chi-squared test article.

In a nutshell, when counts in any of the cells in your table are fewer than 5 then one of the assumptions is broken. I think that's what the error message is referring to. In the article linked you can also find about the correction that can be applied.

-
There are two problems with your less-than-five count rule of thumb. The first is that the correct statement refers to the expected counts rather than the actual counts. The second is that it is too severe. The $\chi^2$ approximation often works well even when a small proportion of expected counts is less than five. In this case, where all the column marginals are five or less, it is obvious that every expected count is small and so we are advised to be cautious. Also, the correction mentioned in the Wikipedia article applies only in the one DF case; this case has 6 DF. –  whuber Jan 7 at 15:01