Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
For a 2x2 contingency table, I understand when the expected frequency is less than 5 that using Fisher's exact test is preferred (I recall reading it is done so that a multinomial is approximately chisq under these conditions).
If the chi squared test was used with an expected frequencies less than 5 - can a statement be made in general that it will under/overestimate the type I or II errors relative to Fisher's test? Is there a paper available which explains this?
Many thanks.
I recall reading it is done so that a multinomial is approximately chisq under these conditions
This is not the case; it's the chi-squared statistic, computed over a bivariate distribution of counts from Poisson, multinomial, product-multinomial, or hypergeometric that is approximately distributed as chi-squared when the expected counts are not too small.
The when sample sizes are 'sufficiently large', the multinomial may be approximated by a degenerate multivariate normal. It's this that leads to an (approximately) chi-squared distribution for the chi-squared statistic.
can a statement be made in general that it will under/overestimate the type I or II errors relative to Fisher's test?
Fisher's test measures association differently from the way a chi-squared statistic does, so it doesn't really make sense to hold it up as the standard against which the Pearson chi-squared test of association should be measured. (In any case, the direct question is that p-values are sometimes larger and sometimes smaller compared to Fisher's exact test)
Under the same conditions as for the Fisher's exact test (i.e. conditioning on the margins -- which are almost ancillary), you can construct an exact test from the chi-squared statistic itself, and that is a standard against which you could reasonably measure the performance of the chi-squared approximation to the distribution of the test statistic under the null.
The exact p-values may either be calculated by complete enumeration or approximated to any desired degree of accuracy via simulation.
Again, it is not the case that the chi-squared approximation always above or below the exact test computed on the same statistic.
In fact I just tried three small tables -- in the first one the ordinary chi-squared test gave the smallest p-value, in the second one Fisher's exact test gave the smallest p-value and in the third the exact test based on the chi-squared statistic gave the smallest p-value; these were the first three cases I tried.
The distribution of the Pearson chi-squared statistic is discrete, but the chi-squared distribution is continuous. The mean and variance are correct even at small samples, it's mainly the discreteness that's the issue.
