6
$\begingroup$

I'm trying to find whether there's a correlation between two variables using Chi Square test and Fisher's Exact test. I get two different p values when I run Chi Square test and Fisher's Exact test. Which value should I take? Take Chi squared p value and do not reject H0 or take Fisher's Exact p value ad reject H0? I ran Fisher's Exact test as well because I get the following error msg when I run Chi Squared test.

Contingency table

enter image description here

> chisq.test(ebtable6)

    Pearson's Chi-squared test

data:  ebtable6
X-squared = 7.3866, df = 2, p-value = 0.02489

Warning message:
In chisq.test(ebtable6) : Chi-squared approximation may be incorrect

> fisher.test(ebtable6)

    Fisher's Exact Test for Count Data

data:  ebtable6
p-value = 0.05018
alternative hypothesis: two.sided
Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

You should not run two different tests. Choose your test first, before you run any tests and preferably before you examine your data values (though it would be permissible to consider the marginal totals).

By looking at both p-values before you decide which to use, you are (quite rightly) open to charges of p-hacking.

Since you're prepared to condition on the margins (you did an exact test after all), you could consider using simulated p-values to deal with the low expected in the (1,1) cell... I just did a million such simulations in R, it only takes a couple of seconds (the chisq.test default of 2000 is a bit small)

(However, you'll still have looked at the p-values... so that issue won't go away)

Improve this answer
$\endgroup$
11
  • $\begingroup$ Hi, Glen! Would you mind sharing the code? $\endgroup$
    – Antoni Parellada
    Commented Feb 16, 2017 at 1:00
  • 2
    $\begingroup$ chisq.test(matrix(c(4,5,23,20,104,496), nrow=3),simulate.p.value = TRUE, B = 10000000) $\endgroup$
    – bdeonovic
    Commented Feb 16, 2017 at 2:28
  • $\begingroup$ @Glen_b: Thank you for the code. The reason I used Fisher's Exact was because to do the Chi square test, 80% of the frequencies should be > 5. Since this is violated, I used Fisher's Exact on this. Is it wrong? I did the same with some other contingency tables and both Chi Square and Fisher's Exact gave the same answer. $\endgroup$
    – user147313
    Commented Feb 16, 2017 at 4:31
  • 1
    $\begingroup$ @sug You get different values for p by simulating, yes, because the chi-squared statistic doesn't actually have a chi-square distribution (they're more likely to differ more when the warning is given). (If you didn't get different values by doing simulation what would be the point in doing it?). Again, you're making the error of computing multiple p-values for the same test and leaving yourself with the problem of choosing ... which leaves you again, in the territory of either (a) cheating or (b) looking like you are or (c) always having to take the highest p-value to avoid the accusation $\endgroup$
    – Glen_b
    Commented Feb 16, 2017 at 7:46
  • 1
    $\begingroup$ That makes sense and is conservative. If you are ambitious you could define a population with proportions as similar to your sample as possible but modified to make the null hypothesis true. Then sample from this population and compute Chi Squared for each sample and determine the proportion of times the null hypothesis is incorrectly rejected at .05. Note that you wouldn't be fixing the marginal frequencies, just the proportions in the population. If the Type I error rate is not inflated a Chi Squared Test for your data is justified. $\endgroup$
    – David Lane
    Commented Feb 16, 2017 at 22:33
1
$\begingroup$

This is not surprising since the tests have different statistical bases. The Fisher Exact test is a randomization test computed assuming both the row and column marginals are fixed (which they very rarely are) and is very conservative when they are not. The Chi Squared test is an approximation but works well in practice, even with sample sizes much smaller than yours. The warning message doesn't make sense since it is a given that an approximation is not exact so it's unclear what is meant by "incorrect." You have expected frequencies less than 5 that some might say is a problem, but simulation studies indicate it is not in most cases. In short, your marginals are not fixed so I recommend you go with the Chi Squared Test.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The warning does make sense, approximation with chi-square only make sense with more than sufficient sample size in each cell. $\endgroup$
    – SmallChess
    Commented Feb 16, 2017 at 3:13
  • 2
    $\begingroup$ Warnings are based on expected frequencies not sample sizes. What doesn't make sense is to say that the approximation may be "incorrect" since it is a given that it is not exact and the meaning of "correct" in this context is hard to figure out. Perhaps it should have said "inaccurate." This study shows that small expected frequencies are rarely a problem. Bradley, D. R., Bradley, T. D., McGrath, S. G., & Cutcomb, S. D. (1979) Type I error rate of the chi square test of independence in r x c tables that have small expected frequencies. Psychological Bulletin, 86, 1200-1297. $\endgroup$
    – David Lane
    Commented Feb 16, 2017 at 4:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.