71
$\begingroup$

I have data showing fire fighter entrance exam results. I am testing the hypothesis that exam results and ethnicity are not mutually independent. To test this, I ran a Pearson chi-square test in R. The results show what I expected, but it gave a warning that "In chisq.test(a) : Chi-squared approximation may be incorrect."

> a
       white black asian hispanic
pass       5     2     2        0
noShow     0     1     0        0
fail       0     2     3        4
> chisq.test(a)

    Pearson's Chi-squared test

data:  a
X-squared = 12.6667, df = 6, p-value = 0.04865

Warning message:
In chisq.test(a) : Chi-squared approximation may be incorrect

Does anyone know why it gave a warning? Is it because I am using a wrong method?

Improve this question
$\endgroup$
1
  • 6
    $\begingroup$ Never trust a result with so few measurements. When you have hundreds of people in each column, then you may have some confidence in your results. Still, the result might be more due to the neighborhood or wealth than to the race itself. $\endgroup$
    – Rodrigo
    Commented Mar 20, 2017 at 1:20

6 Answers 6

Reset to default
62
$\begingroup$

It gave the warning because many of the expected values will be very small and therefore the approximations of p may not be right.

In R you can use chisq.test(a, simulate.p.value = TRUE) to use simulate p values.

However, with such small cell sizes, all estimates will be poor. It might be good to just test pass vs. fail (deleting "no show") either with chi-square or logistic regression. Indeed, since it is pretty clear that the pass/fail grade is a dependent variable, logistic regression might be better.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ What actually simulate.p.value = TRUE does when added to chisq.test? $\endgroup$
    – Al14
    Commented Jun 1, 2017 at 11:12
  • 2
    $\begingroup$ It uses simulations to find the p value $\endgroup$
    – Peter Flom
    Commented Jun 1, 2017 at 11:47
  • 8
    $\begingroup$ Note that simulate.p.value = TRUE uses simulation conditional on the marginals, so is really a version of the Fisher exact test. $\endgroup$
    – kjetil b halvorsen ♦
    Commented Apr 16, 2019 at 11:33
32
$\begingroup$

The issue is that the chi-square approximation to the distribution of the test statistic relies on the counts being roughly normally distributed. If many of the expected counts are very small, the approximation may be poor.

Note that the actual distribution of the chi-square statistic for independence in contingency tables is discrete, not continuous.

The noshow category will be a big contributor to the problem; one thing to consider is merging noshow and fail. You'll still get the warning but it won't affect the results nearly so much and the distribution should be quite reasonable (the rule that's being applied before the warning is given is too strict).

But in any case, if you're willing to condition on the margins (as you do when running Fisher's exact test) you can deal with the problem very easily in R; set the simulate.p.value argument to TRUE; then you aren't reliant on the chi-square approximation to the distribution of the test statistic.

Improve this answer
$\endgroup$
7
  • $\begingroup$ can you please explain to me why "..the chi-square approximation to the distribution of the test statistic relies on the counts being roughly normally distributed"? I do not understand how this can be true if one for example have a 2x2 contingency table. How can the counts be (approximately) normally distributed? How can white, black, hispanic and asian counts possibly be normally distributed? Do you mean just slightly equal? And how does this relate to this question? :stats.stackexchange.com/questions/141407/… $\endgroup$
    – Erosennin
    Commented Mar 18, 2015 at 13:59
  • $\begingroup$ The multivariate distribution of the count random variables needs to be approximately normal (though it will be degenerate). The set of observed counts is only a single vector-observation from this multivariate normal -- you can't judge the distribution from one observation. To make the assessment I'm talking about you need to rely on the assumptions; it's reasonably easy to do it for the individual cells (i.e. the marginal distribution for a given cell, under the null). You seem to be combining counts across cells, but that makes no sense because they all come from different distributions $\endgroup$
    – Glen_b
    Commented Mar 18, 2015 at 14:11
  • $\begingroup$ First, thank you for taking the time! So you are saying that the counts "downwards" the contingency table should be (degenerately) multivariate normal, if we looked at many observations? Would this not mean that counts of each individual cell should be normal as well, and also counts "sideways" the contingency table (I assume this is what you mean with 'across')? F.ex a cell with expected value 5, should be normally distributed around 5, right? So if a cell across has expected value 40, this cell should be normally distributed around 40, and together a multivariate normal of mean 5 and 40, no? $\endgroup$
    – Erosennin
    Commented Mar 18, 2015 at 17:48
  • 1
    $\begingroup$ In the general $r\times c$ case with fixed margins (which is the one I've had in mind), we'd be stacking all the variables within the table into a vector of length $rc$, but they lie in a hyperplane of dimension $(r-1)(c-1)$ -- that's the degeneracy. In the 2x2 case that's 1 degree of freedom, and the 4 cell-counts lie along a line in 4D space. But there's not really space to give proper details. You still haven't quite got it (though you seem to be closer). You might like to repost something like your first question (about the sense in which the values are approximately normal) as a question. $\endgroup$
    – Glen_b
    Commented Mar 19, 2015 at 0:28
  • $\begingroup$ This is heavy, and very intereseting. If you ever have the time, I reposted my first question here: stats.stackexchange.com/questions/142429/… . $\endgroup$
    – Erosennin
    Commented Mar 19, 2015 at 7:01
18
$\begingroup$

For such small counts, you could use Fisher's exact test:

> fisher.test(a)

        Fisher's Exact Test for Count Data

data:  a 
p-value = 0.02618
alternative hypothesis: two.sided
Improve this answer
$\endgroup$
5
$\begingroup$

Please see the "Assumptions" section of Pearson's chi-squared test article.

In a nutshell, when counts in any of the cells in your table are fewer than 5 then one of the assumptions is broken. I think that's what the error message is referring to. In the article linked you can also find about the correction that can be applied.

Improve this answer
$\endgroup$
1
  • 8
    $\begingroup$ There are two problems with your less-than-five count rule of thumb. The first is that the correct statement refers to the expected counts rather than the actual counts. The second is that it is too severe. The $\chi^2$ approximation often works well even when a small proportion of expected counts is less than five. In this case, where all the column marginals are five or less, it is obvious that every expected count is small and so we are advised to be cautious. Also, the correction mentioned in the Wikipedia article applies only in the one DF case; this case has 6 DF. $\endgroup$
    – whuber ♦
    Commented Jan 7, 2014 at 15:01
1
$\begingroup$

Your main question talks about the sample size, but I see that more than two groups are compared. If the p-value from the test is 0.05 or less, it would be difficult to interpret the results. Therefore, I am sharing a brief script that I use in such situations:

# Load the required packages:
library(MASS) # for chisq
library(descr) # for crosstable

CrossTable(a$exam_result, a$ethnicity
       fisher = T, chisq = T, expected = T,
       prop.c = F, prop.t = F, prop.chisq = F, 
       sresid = T, format = 'SPSS')

This code will generate both Pearson's Chi-square and Fisher's Chi square. It produces counts as well as proportions of each of the table entries. Based on the standardised residuals or z-values scores i.e., 

sresid

If it is outside the range |1.96| i.e., less than -1.96 or greater than 1.96, then it is significant p < 0.05. The sign would then indicate whether positively related or negatively.

Improve this answer
$\endgroup$
1
  • $\begingroup$ is there a way to use a different p-value? ex. p < 0.01 or p < 0.1? $\endgroup$
    – Nneka
    Commented Mar 3, 2023 at 16:56
0
$\begingroup$

Your counts per cell are too low. The general rule of thumb is, if the count is bellow 5, use fisher.test. 

> fisher.test(a)

The Fisher exact test extends well to small and large counts, while the chisq.test is generally used for larger counts. You have several values that are 0 and all are below 5, so the Fisher test is what you need!

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.