Chi-square: naïve compensation for low frequencies

Question

As far as I understood it, chi-square provides a measure for determining the similarity of the expected and observed (empirical and theoretical) distributions of nominal variables. It can be employed, for example, in a goodness-of-fit-test that enables to assign a probability to an observed distribution not having been produced by an expected theoretical distribution.

Consider the following example: variable $X $ can be a with a probability of 60%, b with a probability of 30%, and c with a probability of 10%. We may have observed $X $ being a 55 times, being b 21 times, and being c 16 times. The total number of observations is 55 + 21 + 16 = 92. We can then calculate the chi-square statistics as follows.

$\chi^2 = \frac{(55-.6*92)^2}{.6*92} + \frac{(21-.3*92)^2}{.3*92} + \frac{(16-.1*92)^2}{.1*92} $

$\chi^2 = 0.000724638 + 1.578260870 + 5.0260869570 $

$\chi^2 = 6.605072465 $

If we have, however, only a small number of observations, chi-square does not provide reliable results anymore. This is compensated for by Fisher's exact test and Yates's Correction (see Wikipedia). Without compensation, the following problem occurs. Assume our number of observations to be a: 2 times, b: 1 time, and c: 0 times. The total number of observations now is merely 3.

$\chi^2 = \frac{(2-.6*3)^2}{.6*3} + \frac{(1-.3*3)^2}{.3*3} + \frac{(0-.1*3)^2}{.1*3} $

$\chi^2 = 0.022222222 + 0.011111111 + 0.3 $

$\chi^2 = 0.333333333 $

Although the three observations are distributed as closely as possible according to the expected probabilities, we still observe $0 < \chi^2 $. The first part of my question is: Is the impossibility to distribute discrete occurrences exactly according to given probabilities the reason for chi-square's inability to deal with low frequencies?

Following from this is this thought. If the above gives rise to the problems with lower frequencies, can the expected empirical distribution be constructed from integers instead of floating point numbers? In the case above, the expectations for calculating chi-square could be determined as follows.

1. Initially: expected empirical distribution a: 0 times, b: 0 times, c: 0 times
2. Sum of probability differences to expected theoretical distribution: $|.0 - .6| + |.0 - .3| + |.0 - .1| = 1 $.
3. Adding 1 to a changes the differences to $|1. - .6| + |.0 - .3| + |.0 - .1| = .8 $.
4. Adding 1 to b changes the differences to $|0. - .6| + |1. - .3| + |.0 - .1| = 1.4 $.
5. Adding 1 to c changes the differences to $|0. - .6| + |.0 - .3| + |1. - .1| = 1.8 $.
6. One of the three observations is added to a, as it minimizes the difference to the target distribution.
7. The same is repeated until all observations are 'distributed' and we end up with the expected empirical distribution a: 2 times, b: 1 time, c: 0 times.

The chi-square value from comparing a sparse observations with such expectations would be $0 $. The second part of my question is: Is this a valid method and does it enable chi-square to handle low frequencies?

Answer 1

Is the impossibility to distribute discrete occurrences exactly according to given probabilities the reason for chi-square's inability to deal with low frequencies?

No... χ²(k) is the distribution for the sum of the squares of k i.i.d. standard normals. The formula you use to calculate the χ² test statistic is an approximation to this. By the central limit theorem each term in the formula (E-O)²/E, considered as a random variable, is approximately normally distributed, as long as O is actually drawn from the hypothesized distribution and E is large enough for the central limit theorem to apply. For the full proof, see Pearson's theorem in these lecture notes.

Answer 2

Due to my lack of patience (and to provoke corrections), I answer my question with what I could determine so far.

The answer seems to be no in both cases. Although Yates's correction for continuity suggests that the discreteness of low numbers is at least part of the problem (which can be compensated for by subtracting $.5$ from the difference between observed and expected frequency), there can be observed low frequency distributions that match exactly onto the predicted values. Still, however, chi-square is imprecise. This happens, when the division of the product of row marginals and column marginals and the total number of observations is a whole number.

Consider this example (from Wikipedia): Men Women row sum dieting 1 9 10 not dieting 11 3 14 column sum 12 12 24

$\chi^2 = \frac{(1-10*12/24)^2}{10*12/24} + \frac{(11 - 14*12/24)^2}{14*12/24} + \frac{(9 - 10*12/24)^2}{10*12/24}+ \frac{(3 - 14*12/24)^2}{14*12/24} = 10.97143 $

In all four cases, the expected values are whole numbers (i.e., $10*12/24=5 $ and $14*12/24=7 $). In such cases, there is no difference between the classical $\chi^2 $-value and the one proposed above (with whole numbers as expected values), denoted as $\chi_m^2 $ for 'modified'. If integer expectations could increase precision under low frequencies, $\chi^2 $ should not be identical to $\chi_m^2 $.

That these are not special cases, where $\chi^2 $ provides exact results for low frequencies can be seen with Fisher's exact test.

$p_f = \frac{(1+9)!*(11+3)!*(1+11)!*(9+3)!}{1!*9!*11!*3!*24!} = 0.00135 $

Calculating the $\chi^2 $ distribution with a degree of freedom of 1, however, $p_{\chi^2} = p_{\chi_m^2} = 0.00050 $.

Another indicator is that Yates's correction also modifies $\chi^2 $ despite integer expectations. Subtracting $.5 $ from each difference between observed and expected value yields $\chi_Y^2 = 8.4 $ and not $\chi^2 $ or $\chi_m^2$ from above.

Also, $p_{\chi_Y^2} = 0.00206 $, and therefore the difference $|p_f - p_{\chi_m^2}| = 0.00085 $ between the true probability and the one derived from $\chi_m^2 $ is even larger than the one to the probability derived from the Yates's correction: $|p_f - p_{\chi_Y^2}| = 0.00072 $.

Therefore, it can be concluded that $\chi^2 $'s problems with low frequencies do no originate (only) in the impossibility to evenly distribute discrete observations according to continuous expectations.