Sample size for Pearson's chi-square test of independence

Question

I'm trying to do a $\chi^2$ test of independence between two variables. The problem I have is that I'm struggling with the sample size for the test. I always end up with some cells that have 0 samples.

I have the data on the entire population available, but I can't use it for hypothesis testing so I've been trying with various sizes with and without replacement. Also, the population is quite small - only 162.

With smaller sample sizes the chisq() function in R kept coming up with error messages that the estimation may be incorrect.

> chisq = chisq.test(tbl)
Warning message:
In chisq.test(tbl) : Chi-squared approximation may be incorrect

Now I've gone up to a sample size of 100% of population with replacement. The error has disappeared, but I'm concerned since:

a) I still have 0 samples in some cells:

                       var2_high    var2_low    var2_medium    var2_very_high
  var1_high                12           0             10                 3
  var1_low                 10          20              9                 1
  var1_medium               5          23             19                 0
  var1_very_high            9           0              0                41

And b) I'm not sure if such sample size is acceptable.

Can anyone help me with these questions?

Answer 1

You are seeing the message above because the chi squared approximation is unreliable when the sample size is small. I would recommend that you use the original data and perform a Fisher exact test. An example is given below of when this issue might arise and how we can address it using the aforementioned test.

Suppose we have the following sample data below. As an aside, the Fisher exact test was created from the lady tasting tea experiment.

 Truth
Guess  Milk Tea
  Milk    3   1
  Tea     1   3

We want to test the hypothesis that the two variables are independent. Using a chi squared test we get the warning below:

Code:
TeaTasting <- matrix(c(3, 1, 1, 3), 
                     nrow = 2, 
                     dimnames = list(Guess = c("Milk", "Tea"), Truth = c("Milk", "Tea")))

chiSqTest= chisq.test(TeaTasting)

Warning message:
In chisq.test(TeaTasting) : Chi-squared approximation may be incorrect

This isn't surprising since the sample size is relatively small. Furthermore, we see that the expected counts are all less than 5.

chiSqTest$expected

      Truth
Guess  Milk Tea
  Milk    2   2
  Tea     2   2

In this case, we can use a Fisher exact test to test our hypothesis.

fisher.test(TeaTasting)

Given that our p-value is much larger than 0.05, we can conclude that there is no statistical evidence suggesting that the two variables are independent

Fisher's Exact Test for Count Data

data:  TeaTasting
p-value = 0.4857
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
   0.2117329 621.9337505
sample estimates:
odds ratio 
  6.408309