Chi-squared approximation may be incorrect

Question

Please I need your help. I am trying to run this data using R but it throws an error message 'Chi-squared approximation may be incorrect.' Is something wrong with my data?

> Hypotension_tab1 <- table(VR1$'Hypotension during treatment?',VR1$'Maximum tolerated dose of Vericiguat (mg)')
> Hypotension_tab1
     
      2.5 5 7.5 10
  No    5 4   1  1
  Yes   1 4   0  1

> chisq.test(Hypotension_tab1)

    Pearson's Chi-squared test

data:  Hypotension_tab1
X-squared = 2.404, df = 3, p-value = 0.4929

Warning message:
In chisq.test(Hypotension_tab1) :
Chi-squared approximation may be incorrect

Answer 1

The issue is that with small expected values, the chi-squared approximation to the distribution of the test statistic may be poor.

With such small sample size your chances of finding anything were likely to be extremely small (and with very low power, even a rejection of $H_0$ would not necessarily be particularly encouraging that you'd found a real effect).

Leaving the issue of the chi-squared approximation being poor at these sample sizes aside for the moment, let me explain some issues. It likely none of the things I discuss would have helped with the present data, but now you've carried out a test on the data already (and presumably would have happily gone about your business if you'd found a significant result and no warnings were issued), there's really nothing more you could likely do here without being accused of cherry-picking/p-hacking. Nevertheless this discussion may help for next time:

(i) Your dose variable is ordered, and indeed is literally numeric. Consequently, one might well expect that you'd have a working hypothesis that reflects that ordering and in turn a test that makes use of it. Ignoring the ordering in such circumstances is throwing away power. Indeed the working hypothesis is probably even directional -- though perhaps you're working in an area that doesn't tolerate taking advantage of that directionality even in a pre-registered analysis. In any case making use of the fact that the dose variable is not nominal would be important especially at such low sample sizes.

(ii) The issue with the distribution of the chi-squared statistic and indeed with the Fisher exact test and the G-test and other alternatives on a nominal/nominal contingency table is that with such small samples the exact distributions for each are highly discrete.

One of the resulting issues is that there are few available significance levels. Consequently, if one simply ploughs ahead willy-nilly and compares p-values to an arbitrarily chosen significance level (in particular, one that's chosen without regard to the available significance levels given the margins), the test may be highly conservative, with all that implies for power. In some situations (that I have seen in practice) there is literally no available significance level less than $\alpha$, and in that case you have no chance to see a rejection at all.

Something can be done about that discreteness of available significance levels, which is of practical help at least some of the time; one can use a second suitable test (and a third in its turn, if the second is also discrete) to break ties and so generate a finer set of available significance levels. In some cases that's quite valuable.

It turns out not to be such a major issue for your table, at least for an exact test using the chi-squared statistic: you have an available significance level that's only moderately conservative (around 0.048) for a 5% test using an exact test based on the chi-squared statistic (I'm presuming that's what you were after). It's a bit worse for the Fisher exact test; it looks like the highest significance level below 0.05 is about 0.037 in its case (and in that situation a tie-breaker would be useful). I would investigate the G-test as well but I expect the story is not much different from the chi-squared test in this case.

On the other hand, if you did a chi-squared test and ignored the warnings, you'd actually be operating at a considerably worse significance level even than the Fisher test.

---

If you are aware you're likely to have very low sample sizes, it's really important to investigate the ways to make the best possible use of what data you get, before you collect the data.

Answer 2

Fisher's is primarily designed for a 2x2 table but for this example it should work fine. You could also use simulate.p.value = FALSE in chisq.test() which will generate 2000 tables with the same marginal values (row and column sums) to compare with your observed data. Both approaches show that the original Chi Square p-value is an underestimate:

chisq.test(tbl)
# 
#   Pearson's Chi-squared test
# 
# data:  tbl
# X-squared = 2.404, df = 3, p-value = 0.4929
fisher.test(tbl)
# 
#   Fisher's Exact Test for Count Data
# 
# data:  tbl
# p-value = 0.5984
# alternative hypothesis: two.sided
chisq.test(tbl, simulate.p.value=TRUE)
# 
#   Pearson's Chi-squared test with simulated p-value (based on 2000 replicates)
# 
# data:  tbl
# X-squared = 2.404, df = NA, p-value = 0.6802

Notice that the last approach gives the most conservative estimate of the p-value. Since none of them approach .05. The best conclusion is that the sample is too small to suggest a significance difference.

Answer 3

First it is not an error but a warning.

Second, as a general rule the count of individuals per bin should be greater than 5 for the chi sq test. Most of your bins have only 0 or 1 in them. So either add multiple bins together to exceed 5 or use another test such as the Fisher's Exact test: fisher.test().