What is the intuition behind pooling frequencies that are less than five in a Chi-squared goodness of fit test?

Even though I have been told that it is not used in industries today, the concept is still taught at the undergraduate level. And is incorporated in every question and practical of goodness of fit test that we solve. However I have never come across any convincing explanation either intuitively or mathematically as to why we do so. Can you help me on this?

  • 1
    $\begingroup$ Technically this is wrong (although its validity can be checked, so when appropriately done it can produce useful information). See stats.stackexchange.com/a/17148/919 for an account of what is required for the chi-squared test to apply. $\endgroup$
    – whuber
    Aug 30 '18 at 17:05
  • $\begingroup$ @whuber the answer you referred me to was regarding the degrees of freedom for a chi-square statistic. But my question is regarding the problems caused by the presence of low expected frequencies in the table. Why is it necessary to pool those frequencies so as to make them >5? $\endgroup$ Aug 30 '18 at 18:25
  • 1
    $\begingroup$ It sounds like you didn't read that answer: it quite clearly and explicitly addresses your concern. $\endgroup$
    – whuber
    Aug 30 '18 at 18:27
  • $\begingroup$ @whuber Yes I did read the whole answer. I am sorry I am not getting what part of that answer you are exactly referring to? $\endgroup$ Aug 30 '18 at 18:29
  • 1
    $\begingroup$ In somewhat the same sense as a binomial distribution with $n$ sufficiently large and $p$ sufficiently far from $0$ or $1$ can be approximated by a normal distribution, a (so-called) chi-squared GOF statistic with all expected counts greater than 5 can be approximated by a chi-squared distribution.// If there are many 'cells' some authors say it's OK if most expected counts exceed 5 and all exceed 3. In any case, it's not a fixed rule, it's pretty good guidance based on experience. // Some software (including R) will simulate a P-value in case of too many low expected counts. $\endgroup$
    – BruceET
    Aug 31 '18 at 3:36

After reading @whuber's impressive and comprehensive link, it occurs to me that an explanation focused only on the issue of needing expected counts to be 5 or greater might be helpful to you.

Testing whether a Roulette wheel is fair. Suppose I'm used to betting on colors at US roulette. Out of 38 slots around the wheel there are 2 green, 18 red and 18 black. I'm suspicious that the wheel is biased so that the probabilities are not really 2/38, 18/38, and 18/38.

An inadequate experiment. I decide to watch 20 spins of the wheel and try a chi-squared test on the results. The expected counts for green, red, and black would be $E_1 = 20(1/19),\,$ $E_2 = E_3 = 20(9/19),$ respectively. Notice that $E_1 = 1.053$ is much too small. If $X_1, X_2, X_3$ are the actual observed counts of the three colors in 20 spins, then the chi-squared statistic is $$Q = \sum_{i=1}^3 \frac{(X_i - E_i)^2}{E_i}.$$ $Q$ would have approximately the distribution $\mathsf{Chisq}(df = 2)$ under the null hypothesis that the wheel is fair, provided all of the $E_i$'s were large enough. But in this situation the approximation is not good.

In practice, the distribution of $Q$ does not need to fit the chi-squared distribution throughout the support of $Q.$ We use only right-tail probabilities to find critical values and to decide whether P-values are small enough to reject the null hypothesis.

In a simulation with a million 20-spin experiments, I found that the tail probabilities are not quite right. Under $H_0,$ according to $\mathsf{Chisq}(2),$ 95% of the Q-values should be less than 5.9915. But the actual 95th percentile of the Q-values was about 6.39.

An experiment in which the statistic has chi-squared distribution. By contrast, if I decide to watch 100 spins of the wheel, then $E_1 = 11.11 > 5,$ so the guideline for $Q$ to be chi-squared distributed is met. In another simulation with a million 100-spin experiments the 95th percentile of the Q-values was 5.9914, which agrees with the value from $\mathsf{Chisq}(2)$ within simulation error.

Histograms from the two simulations are shown below, along with the density function of $\mathsf{Chisq}(2).$ It seems clear that, in the case with 20-spins and an expected count that is too small, the so-called chi-squared test statistic does not have a chi-squared distribution.

enter image description here

Note: In order to make a compact figure, both histograms were truncated at 20.

  • $\begingroup$ In the 20-spin test, I could combine green and red, to have categories 'black' and 'not black'. Then there would be good fit to chi-sq, but (a) it wouldn't be the test I want and (b) that wouldn't cure the fundamental problem that 20 spins don't give me much information. $\endgroup$
    – BruceET
    Aug 31 '18 at 9:08
  • $\begingroup$ Is there any mathematical explanation besides this empirical one as to why this happens with low expected counts? $\endgroup$ Aug 31 '18 at 9:09
  • $\begingroup$ I think that would be a little messy. This kind of goodness-of-fit test isn't the likelihood ratio test (which also has an asymptotically chi-squared distribution). // A trivial observation is that if $E_i$ is small, the term $(X_i-E_i)^2/E_i$ of $Q$ has the potential to be larger than appropriate. If nothing else, that would make $Q$ even "more discrete" than it already is. $\endgroup$
    – BruceET
    Aug 31 '18 at 9:19
  • $\begingroup$ Because of the difference (Oi - Ei) or the denominator Ei? $\endgroup$ Aug 31 '18 at 9:24
  • $\begingroup$ Denominator is main concern. $\endgroup$
    – BruceET
    Aug 31 '18 at 9:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.