# Warning in R - Chi-squared approximation may be incorrect

I have data showing fire fighter entrance exam results. I am testing the hypothesis that exam results and ethnicity are not mutually independent. To test this, I ran a Pearson chi-square test in R. The results show what I expected, but it gave a warning that "In chisq.test(a) : Chi-squared approximation may be incorrect."

> a
white black asian hispanic
pass       5     2     2        0
noShow     0     1     0        0
fail       0     2     3        4
> chisq.test(a)

Pearson's Chi-squared test

data:  a
X-squared = 12.6667, df = 6, p-value = 0.04865

Warning message:
In chisq.test(a) : Chi-squared approximation may be incorrect


Does anyone know why it gave a warning? Is it because I am using a wrong method? Thanx in advance.

-

It gave the warning because many of the expected values will be very small and therefore the approximations of p may not be right.

In R you can use chisq.test(a, simulate.p.value = TRUE) to use simulate p values.

However, with such small cell sizes, all estimates will be poor. It might be good to just test pass vs. fail (deleting "no show") either with chi-square or logistic regression. Indeed, since it is pretty clear that the pass/fail grade is a dependent variable, logistic regression might be better.

-
can you please show how can i apply logistic regression ? –  user1883491 Jan 7 '14 at 12:39
I haven't time right now to do so in detail but look at glm and search on logistic regression. –  Peter Flom Jan 7 '14 at 12:49

The issue is that the chi-square approximation to the distribution of the test statistic relies on the counts being roughly normally distributed. If many of the expected counts are very small, the approximation may be poor.

Note that the actual distribution of the chi-square statistic for independence in contingency tables is discrete, not continuous.

The noshow category will be a big contributor to the problem; one thing to consider is merging noshow and fail. You'll still get the warning but it won't affect the results nearly so much and the distribution should be quite reasonable (the rule that's being applied before the warning is given is too strict).

But in any case, if you're willing to condition on the margins (as you do when running Fisher's exact test) you can deal with the problem very easily in R; set the simulate.p.value argument to TRUE; then you aren't reliant on the chi-square approximation to the distribution of the test statistic.

-
can you please explain to me why "..the chi-square approximation to the distribution of the test statistic relies on the counts being roughly normally distributed"? I do not understand how this can be true if one for example have a 2x2 contingency table. How can the counts be (approximately) normally distributed? How can white, black, hispanic and asian counts possibly be normally distributed? Do you mean just slightly equal? And how does this relate to this question? :stats.stackexchange.com/questions/141407/… –  Eros Ram Mar 18 at 13:59
The multivariate distribution of the count random variables needs to be approximately normal (though it will be degenerate). The set of observed counts is only a single vector-observation from this multivariate normal -- you can't judge the distribution from one observation. To make the assessment I'm talking about you need to rely on the assumptions; it's reasonably easy to do it for the individual cells (i.e. the marginal distribution for a given cell, under the null). You seem to be combining counts across cells, but that makes no sense because they all come from different distributions –  Glen_b Mar 18 at 14:11
First, thank you for taking the time! So you are saying that the counts "downwards" the contingency table should be (degenerately) multivariate normal, if we looked at many observations? Would this not mean that counts of each individual cell should be normal as well, and also counts "sideways" the contingency table (I assume this is what you mean with 'across')? F.ex a cell with expected value 5, should be normally distributed around 5, right? So if a cell across has expected value 40, this cell should be normally distributed around 40, and together a multivariate normal of mean 5 and 40, no? –  Eros Ram Mar 18 at 17:48
In the general $r\times c$ case with fixed margins (which is the one I've had in mind), we'd be stacking all the variables within the table into a vector of length $rc$, but they lie in a hyperplane of dimension $(r-1)(c-1)$ -- that's the degeneracy. In the 2x2 case that's 1 degree of freedom, and the 4 cell-counts lie along a line in 4D space. But there's not really space to give proper details. You still haven't quite got it (though you seem to be closer). You might like to repost something like your first question (about the sense in which the values are approximately normal) as a question. –  Glen_b Mar 19 at 0:28
This is heavy, and very intereseting. If you ever have the time, I reposted my first question here: stats.stackexchange.com/questions/142429/… . –  Eros Ram Mar 19 at 7:01

For such small counts, you could use Fisher's exact test:

> fisher.test(a)

Fisher's Exact Test for Count Data

data:  a
p-value = 0.02618
alternative hypothesis: two.sided

-

Please see the "Assumptions" section of Pearson's chi-squared test article.

In a nutshell, when counts in any of the cells in your table are fewer than 5 then one of the assumptions is broken. I think that's what the error message is referring to. In the article linked you can also find about the correction that can be applied.

-
There are two problems with your less-than-five count rule of thumb. The first is that the correct statement refers to the expected counts rather than the actual counts. The second is that it is too severe. The $\chi^2$ approximation often works well even when a small proportion of expected counts is less than five. In this case, where all the column marginals are five or less, it is obvious that every expected count is small and so we are advised to be cautious. Also, the correction mentioned in the Wikipedia article applies only in the one DF case; this case has 6 DF. –  whuber Jan 7 '14 at 15:01