0
$\begingroup$

Imagine a 6-sided die. I roll it N times and calculate the Chi-Square statistic on the 6-entry table of frequencies of these rolls.

In my program, I calculate the chi-square statistic as follows:

$\sum_i (O_i-E_i)^2/E_i^2$

The distribution of this statistic is supposed to follow a Chi-Square distribution with DOF 5. The theoretical distribution of the above statistic, as far as I can tell from the literature on using Pearson's Chi-Square Test to test fit to a distribution, is not sensitive to the number of samples/rolls (number N).

However, in practice, it is sensitive to N.

Therefore, the Chi-Square Distribution cannot model this statistic for more than one particular value of N (but we do not specify N when we specify the distribution, only DOF which is related to the number of free variables specifying frequencies).

Yes, I do not know deeply the theoretical foundation of Chi-Square so I am looking for a way of understanding this seeming paradox.

I can put this question another way.

In this code, I have a simulation that I run M times (fixed high M e.g. 10,000). In each simulation I simulate rolling a fair die N times to get N samples and calculate the Chi-Square statistic. For each different choice of N I get a different histogram of the Chi Square statistic (similar shape, different scale). I also sample from the theoretical Chi-Square distribution with DOF 5 (M times) and get the same shape but different values. The 95th percentile of the samples from the theoretical distribution is around 11. From the empirical distributions, for N=200 it is around 1.5 and for N=5000 it is around 1.2 (this decreasing of the threshold would seem appropriate as I am increasing the number of samples from which my statistic is calculated, but certainly is not approaching the theoretical value). Obviously, if I were to use the theoretical distribution against an experiment with N=200 I would be overestimating the threshold that would be required to make a type-1 error 5% of the time when the die is fair (recalling the aim is to check fit to distribution - in this case a uniform). As far as I can tell, none of this converges to the theoretical distribution as either N or M increase.

Where is my fundamental mistake?

Thank you in advance for your feedback.

There was no fundamental mistake. Thanks to Glen_b's own simulation which gave theoretically appropriate results, I could see that there must have been a problem with the code, which there was. The primary problem was not the formula. Simply, the code was sampling from a 5 sided die where it should have been from a 6 sided die.

|cite|improve this question
$\endgroup$
  • $\begingroup$ 1. The distribution of the chi-square statistic does depend on N. Have you seen the widely-used "rule of thumb" that the expected value in each cell should be greater than 5? (or any of a number of similar rules) ... Such rules exist because the chi-squared approximation for the distribution of the statistic is not very good when there are small expected values, but it's better as $N$ increases 2. However, if I follow what you're saying correctly you have some other issue that the one I am describing. 3. It's impossible to tell from the infor in the Q what it is you're doing wrong; ... $\endgroup$ – Glen_b May 9 '18 at 15:47
  • $\begingroup$ ... and your link doesn't work -- it wants me to log in. In any case, could you perhaps explain what you're trying to do? $\endgroup$ – Glen_b May 9 '18 at 15:57
  • $\begingroup$ Sorry about the link. Fixed that. From my experiments, as I increase N, the 95% threshold actually goes down (from 1.5 when N=200) but the theoretical threshold for 5 DOF is 11.1 (p=0.05). $\endgroup$ – D.J.Duff May 9 '18 at 16:32
  • $\begingroup$ But I take your point that the threshold calculated from the empirical distribution does seem to converge as N grows. Thank you. $\endgroup$ – D.J.Duff May 9 '18 at 16:33
  • $\begingroup$ In the die example, if I set N to 6, that is I calculate Chi-Square from 6 throws, then the 95th percentile is generally about 12 which is the closest I got to the theoretical threshold for p=0.05. Could it be that when using the Chi-Square distribution for Pearson's Chi-Square test, we assume the number of bins and the sum of the frequencies in the bins to be roughly equal? Because if I increase N, as I wrote, the real threshold gets too low to be comparable to the theoretical distribution. $\endgroup$ – D.J.Duff May 9 '18 at 19:43
1
$\begingroup$

Your formula for the chi-squared statistic is wrong.

It should be $\sum_i (O_i-E_i)^2/E_i$.

When you see surprising or unexpected behavior, it's the obvious thing to check.

With a correct implementation of the statistic you should see a progression something like this:

histograms of chi-squared statistic for the described experiment and the chi-squared density with five degrees of freedom

Here we see histograms for your experiment compared to the $\chi^2_5$ density with N = 25, 50, 100, 200

As you see, things are much as one might expect -- the $\chi^2$ approximation is rough at N=25 ($E_i$ approximately 4 in each cell) and considerably better at $N=200$ (where $E_i$ is approximately 33 in each cell).

In case anyone feels the need to see it, here's my code (done in R) for N=200:

X2.200 = replicate(10000,
                chisq.test(table(factor(sample(6,200,replace=TRUE),levels=1:6)))$statistic
           )

This is not especially nicely organized, but okay for a quick simulation (I actually had this as a single line but it's a couple of characters too long for the window).

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thank you for editing my question to add the formula on the basis of my code, and thank you for pointing that out. I do appreciate your help. Unfortunately, that was only a change that I made to try and achieve convergence to the theoretical. Reverting my code to the non-squared denominator I get a growth in the 95th percentile of chi-square statistic, way beyond the theoretical (N=7 gives me 11, N=200 gives 51, N=5000 gives 1011). $\endgroup$ – D.J.Duff May 10 '18 at 3:45
  • $\begingroup$ So I'll update the formula in the question. Thank you for adding it. $\endgroup$ – D.J.Duff May 10 '18 at 3:46
  • $\begingroup$ If you change the formula in the question, it won't correspond to your code. You should fix your code instead and see if this fixes your problem $\endgroup$ – Glen_b May 10 '18 at 4:24
  • $\begingroup$ Thanks Glen. I changed the code before I changed the question. The link should reflect the fixed code. $\endgroup$ – D.J.Duff May 10 '18 at 4:27
  • $\begingroup$ The only change in the code was to the was to the Chi-Square formula. $\endgroup$ – D.J.Duff May 10 '18 at 4:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.