2
$\begingroup$

For a 2x2 contingency table, I understand when the expected frequency is less than 5 that using Fisher's exact test is preferred (I recall reading it is done so that a multinomial is approximately chisq under these conditions).

If the chi squared test was used with an expected frequencies less than 5 - can a statement be made in general that it will under/overestimate the type I or II errors relative to Fisher's test? Is there a paper available which explains this?

Many thanks.

$\endgroup$

1 Answer 1

3
$\begingroup$

I recall reading it is done so that a multinomial is approximately chisq under these conditions

This is not the case; it's the chi-squared statistic, computed over a bivariate distribution of counts from Poisson, multinomial, product-multinomial, or hypergeometric that is approximately distributed as chi-squared when the expected counts are not too small.

The when sample sizes are 'sufficiently large', the multinomial may be approximated by a degenerate multivariate normal. It's this that leads to an (approximately) chi-squared distribution for the chi-squared statistic.

can a statement be made in general that it will under/overestimate the type I or II errors relative to Fisher's test?

Fisher's test measures association differently from the way a chi-squared statistic does, so it doesn't really make sense to hold it up as the standard against which the Pearson chi-squared test of association should be measured. (In any case, the direct question is that p-values are sometimes larger and sometimes smaller compared to Fisher's exact test)

Under the same conditions as for the Fisher's exact test (i.e. conditioning on the margins -- which are almost ancillary), you can construct an exact test from the chi-squared statistic itself, and that is a standard against which you could reasonably measure the performance of the chi-squared approximation to the distribution of the test statistic under the null.

The exact p-values may either be calculated by complete enumeration or approximated to any desired degree of accuracy via simulation.

Again, it is not the case that the chi-squared approximation always above or below the exact test computed on the same statistic.

In fact I just tried three small tables -- in the first one the ordinary chi-squared test gave the smallest p-value, in the second one Fisher's exact test gave the smallest p-value and in the third the exact test based on the chi-squared statistic gave the smallest p-value; these were the first three cases I tried.

The distribution of the Pearson chi-squared statistic is discrete, but the chi-squared distribution is continuous. The mean and variance are correct even at small samples, it's mainly the discreteness that's the issue.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.