# Fisher Exact / Chi Squared Type I/II for low expected counts

For a 2x2 contingency table, I understand when the expected frequency is less than 5 that using Fisher's exact test is preferred (I recall reading it is done so that a multinomial is approximately chisq under these conditions).

If the chi squared test was used with an expected frequencies less than 5 - can a statement be made in general that it will under/overestimate the type I or II errors relative to Fisher's test? Is there a paper available which explains this?

Many thanks.

I recall reading it is done so that a multinomial is approximately chisq under these conditions

This is not the case; it's the chi-squared statistic, computed over a bivariate distribution of counts from Poisson, multinomial, product-multinomial, or hypergeometric that is approximately distributed as chi-squared when the expected counts are not too small.

The when sample sizes are 'sufficiently large', the multinomial may be approximated by a degenerate multivariate normal. It's this that leads to an (approximately) chi-squared distribution for the chi-squared statistic.

can a statement be made in general that it will under/overestimate the type I or II errors relative to Fisher's test?

Fisher's test measures association differently from the way a chi-squared statistic does, so it doesn't really make sense to hold it up as the standard against which the Pearson chi-squared test of association should be measured. (In any case, the direct question is that p-values are sometimes larger and sometimes smaller compared to Fisher's exact test)

Under the same conditions as for the Fisher's exact test (i.e. conditioning on the margins -- which are almost ancillary), you can construct an exact test from the chi-squared statistic itself, and that is a standard against which you could reasonably measure the performance of the chi-squared approximation to the distribution of the test statistic under the null.

The exact p-values may either be calculated by complete enumeration or approximated to any desired degree of accuracy via simulation.

Again, it is not the case that the chi-squared approximation always above or below the exact test computed on the same statistic.

In fact I just tried three small tables -- in the first one the ordinary chi-squared test gave the smallest p-value, in the second one Fisher's exact test gave the smallest p-value and in the third the exact test based on the chi-squared statistic gave the smallest p-value; these were the first three cases I tried.

The distribution of the Pearson chi-squared statistic is discrete, but the chi-squared distribution is continuous. The mean and variance are correct even at small samples, it's mainly the discreteness that's the issue.