I'm trying to do a $\chi^2$ test of independence between two variables. The problem I have is that I'm struggling with the sample size for the test. I always end up with some cells that have 0 samples.

I have the data on the entire population available, but I can't use it for hypothesis testing so I've been trying with various sizes with and without replacement. Also, the population is quite small - only 162.

With smaller sample sizes the chisq() function in R kept coming up with error messages that the estimation may be incorrect.

> chisq = chisq.test(tbl)
Warning message:
In chisq.test(tbl) : Chi-squared approximation may be incorrect

Now I've gone up to a sample size of 100% of population with replacement. The error has disappeared, but I'm concerned since:

a) I still have 0 samples in some cells:

                       var2_high    var2_low    var2_medium    var2_very_high
  var1_high                12           0             10                 3
  var1_low                 10          20              9                 1
  var1_medium               5          23             19                 0
  var1_very_high            9           0              0                41

And b) I'm not sure if such sample size is acceptable.

Can anyone help me with these questions?

  • 1
    $\begingroup$ The following error happens when the expected value of the cells < 5. A fisher exact test can be used to test if the rows and columns are independent when the expected value of the cells are less than 5. $\endgroup$ – nwaldo Aug 18 at 2:51
  • $\begingroup$ Thanks! I'm a little confused since I still have less than 5 in several cells, but the error has disappeared. $\endgroup$ – pavel Aug 18 at 2:54
  • 2
    $\begingroup$ Just to note, the expected counts are different from the observed values see the following example. I would also recommend that you use the fisher exact test on your original data instead of resampling until the error goes away. The fisher exact test is supposed to be used when the sample size is small see here $\endgroup$ – nwaldo Aug 18 at 3:16

You are seeing the message above because the chi squared approximation is unreliable when the sample size is small. I would recommend that you use the original data and perform a Fisher exact test. An example is given below of when this issue might arise and how we can address it using the aforementioned test.

Suppose we have the following sample data below. As an aside, the Fisher exact test was created from the lady tasting tea experiment.

Guess  Milk Tea
  Milk    3   1
  Tea     1   3

We want to test the hypothesis that the two variables are independent. Using a chi squared test we get the warning below:

TeaTasting <- matrix(c(3, 1, 1, 3), 
                     nrow = 2, 
                     dimnames = list(Guess = c("Milk", "Tea"), Truth = c("Milk", "Tea")))

chiSqTest= chisq.test(TeaTasting)
Warning message:
In chisq.test(TeaTasting) : Chi-squared approximation may be incorrect

This isn't surprising since the sample size is relatively small. Furthermore, we see that the expected counts are all less than 5.

Guess  Milk Tea
  Milk    2   2
  Tea     2   2

In this case, we can use a Fisher exact test to test our hypothesis.


Given that our p-value is much larger than 0.05, we can conclude that there is no statistical evidence suggesting that the two variables are independent

Fisher's Exact Test for Count Data

data:  TeaTasting
p-value = 0.4857
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
   0.2117329 621.9337505
sample estimates:
odds ratio 
| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.