Imagine a 6-sided die. I roll it N times and calculate the Chi-Square statistic on the 6-entry table of frequencies of these rolls.
In my program, I calculate the chi-square statistic as follows:
$\sum_i (O_i-E_i)^2/E_i^2$
The distribution of this statistic is supposed to follow a Chi-Square distribution with DOF 5. The theoretical distribution of the above statistic, as far as I can tell from the literature on using Pearson's Chi-Square Test to test fit to a distribution, is not sensitive to the number of samples/rolls (number N).
However, in practice, it is sensitive to N.
Therefore, the Chi-Square Distribution cannot model this statistic for more than one particular value of N (but we do not specify N when we specify the distribution, only DOF which is related to the number of free variables specifying frequencies).
Yes, I do not know deeply the theoretical foundation of Chi-Square so I am looking for a way of understanding this seeming paradox.
I can put this question another way.
In this code, I have a simulation that I run M times (fixed high M e.g. 10,000). In each simulation I simulate rolling a fair die N times to get N samples and calculate the Chi-Square statistic. For each different choice of N I get a different histogram of the Chi Square statistic (similar shape, different scale). I also sample from the theoretical Chi-Square distribution with DOF 5 (M times) and get the same shape but different values. The 95th percentile of the samples from the theoretical distribution is around 11. From the empirical distributions, for N=200 it is around 1.5 and for N=5000 it is around 1.2 (this decreasing of the threshold would seem appropriate as I am increasing the number of samples from which my statistic is calculated, but certainly is not approaching the theoretical value). Obviously, if I were to use the theoretical distribution against an experiment with N=200 I would be overestimating the threshold that would be required to make a type-1 error 5% of the time when the die is fair (recalling the aim is to check fit to distribution - in this case a uniform). As far as I can tell, none of this converges to the theoretical distribution as either N or M increase.
Where is my fundamental mistake?
Thank you in advance for your feedback.
There was no fundamental mistake. Thanks to Glen_b's own simulation which gave theoretically appropriate results, I could see that there must have been a problem with the code, which there was. The primary problem was not the formula. Simply, the code was sampling from a 5 sided die where it should have been from a 6 sided die.