Timeline for Different p-values for Chi squared and Fisher's Exact R
Current License: CC BY-SA 3.0
13 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jun 23, 2017 at 5:18 | vote | accept | CommunityBot | ||
| Feb 16, 2017 at 22:33 | comment | added | David Lane | That makes sense and is conservative. If you are ambitious you could define a population with proportions as similar to your sample as possible but modified to make the null hypothesis true. Then sample from this population and compute Chi Squared for each sample and determine the proportion of times the null hypothesis is incorrectly rejected at .05. Note that you wouldn't be fixing the marginal frequencies, just the proportions in the population. If the Type I error rate is not inflated a Chi Squared Test for your data is justified. | |
| Feb 16, 2017 at 9:36 | comment | added | Glen_b | Generally the warning is given more often than necessary (as David points out above); If your smallest expected values are not substantially smaller than 5 it's usually fine to use the chi-square. | |
| Feb 16, 2017 at 8:08 | comment | added | user147313 | @Glen_b: Thank you for the clarification :). I think I should take simulated p when the warning is given. | |
| Feb 16, 2017 at 7:51 | comment | added | Glen_b | So for anything that matters, pick a single statistic, and a single way of finding the p-value, and then work out what that p-value is | |
| Feb 16, 2017 at 7:46 | comment | added | Glen_b | @sug You get different values for p by simulating, yes, because the chi-squared statistic doesn't actually have a chi-square distribution (they're more likely to differ more when the warning is given). (If you didn't get different values by doing simulation what would be the point in doing it?). Again, you're making the error of computing multiple p-values for the same test and leaving yourself with the problem of choosing ... which leaves you again, in the territory of either (a) cheating or (b) looking like you are or (c) always having to take the highest p-value to avoid the accusation | |
| Feb 16, 2017 at 7:44 | comment | added | user147313 | Is it possible to get two different values for p when simulated and not? For another contingency table, I get p = 0.007995 when not simulated and = 0.06747 with default B and p = 0.06633 with B= 10000000. Does this suggests there is no relationship between the variables? | |
| Feb 16, 2017 at 7:29 | comment | added | Glen_b | @David Yes, thanks I'm aware. Indeed in this answer I said "numerous studies since have argued that it's too conservative". However, here the lowest expected gets down toward 1 and the simulation I suggested has a fairly different p-value than the ordinary chi-squared one does, so I decided it was not reasonable to say "it probably doesn't matter" when here it seems as if it actually may make a difference. | |
| Feb 16, 2017 at 4:42 | comment | added | David Lane | That rule about expected frequencies has been known to be very conservative for a long time. See Bradley, D. R., Bradley, T. D., McGrath, S. G., & Cutcomb, S. D. (1979) Type I error rate of the chi square test of independence in r x c tables that have small expected frequencies. Psychological Bulletin, 86, 1200-1297. | |
| Feb 16, 2017 at 4:31 | comment | added | user147313 | @Glen_b: Thank you for the code. The reason I used Fisher's Exact was because to do the Chi square test, 80% of the frequencies should be > 5. Since this is violated, I used Fisher's Exact on this. Is it wrong? I did the same with some other contingency tables and both Chi Square and Fisher's Exact gave the same answer. | |
| Feb 16, 2017 at 2:28 | comment | added | bdeonovic | chisq.test(matrix(c(4,5,23,20,104,496), nrow=3),simulate.p.value = TRUE, B = 10000000) | |
| Feb 16, 2017 at 1:00 | comment | added | Antoni Parellada | Hi, Glen! Would you mind sharing the code? | |
| Feb 16, 2017 at 0:24 | history | answered | Glen_b | CC BY-SA 3.0 |