References for excess power of a Chi-Squared test

I am working on an example for Benford's law and using UK house numbers I get the following empirical and theoretical distribution for the first digits. Although the graphical inspection clearly shows that the numbers do conform to Benford's law quite perfectly. I can safely reject the null hypothesis that both distributions are the same, which is of course caused by the large sample size of 93,087 house numbers.

Pearson's Chi-squared test
X-squared = 106, df = 8, p-value <2e-16


I know that this is caused by the large sample size and I find some information for that in Nigrini (2012: p. 154)Google Book Link

I would like to know if there are some further references discussing this problem. Perhaps also some additional reference for the hypotheses raised by Nigrini that this starts with sample sizes larger than 5,000.

Chi-squared seems inappropriate to your aim, which is to determine if there is a strong relationship between your model and your data. Chi squared tests for differences, not similarity nor association so is the inverse of your needs. You cannot 'prove' a null hypothesis, only fail to reject it or state that you have a defined level of confidence it does not apply. Many articles and discussion boards cover this topic extensively, resources I found invaluable when I began to understand this aspect of statistical inference (regretfully more years into my research career than I care to admit, it took me a lot of head aches and diving deeper than I ever wished)

This means that a low p value for your Chi Square test would be interpreted that the probability of there being zero difference between your proposed distribution and you observed distribution is very low. A high p-value would be interpreted as the differences between the distributions could arise from random chance.

Neither of these interpretations are identical to 'graphical inspection clearly shows that the numbers do conform to Benford's law quite perfectly'. This phrase suggests that you wish to demonstrate concordance/equivalence or association between your expected and observed data. This suggests that the hypothesis you wish to falsify is 'the observed data is equivalent to Benford's law within a confidence interval of X'.

Correlation would be a more relevant statistic and will tell you the strength of agreement between your model and the data. $R^2$ will tell you the proportion of your data that can be explained by your model. There will be some that isn't, which suggests that some other factors are perturbing the data slightly away from your ideal model. Your observed data will not be Benfords law plus true random noise, instead it will be Benford's law plus other subtle but real effects that cause small but real deviances. Using a big sample number gives you greater power for detecting these non-random subtle effects. Whether those subtle deviances are big enough to matter will be handled by $R^2$.

Whether it is worth examining the additional factors that mean your data deviates from the model will depend on $R^2$, which I suspect will be large. It will come down to matter of practical significance rather than statistical significance.

It is not that chi squared is over powered, I suggest it is answering a different question to what you want.

• It's not a comparsion between a model and data. It's a comparison between an expected distribution and an empirical observable distribution. I think that's exactly the question to answer with a Chi-Square test.I don't see how there should be any relation to correlation or anything related to a model, because that's not what I am trying to show. Mar 19 '18 at 22:30
• In your case you have modelled your expected distribution using a model (Benford distribution). An expected distribution is a model by definition. It may be an empirically derived one (i.e. a counted distribution of a comparator population), but the point is that you use that to model the population of interest. Could you please spell out the exact null hypothesis that you wish to test so that the answer can be tailored. 'Although the graphical inspection clearly shows that the numbers do conform to Benford's law quite perfectly' is a visual description of the correlation. Mar 20 '18 at 8:36
• What Chi squared is measuring is the residual after you account for the expected distribution (so you would plot the difference between the model (which is what your expected distribution is) and the data on the Y axis. If you plot confidence intervals on such a difference plot then you will have a visual approximation of the Chi-square. I advise you to update your question with this new figure and also to add confidence intervals at each point in the existing figure. That way I can tailor the answer better to your data. Mar 20 '18 at 8:39
• Well, yes, but in my opinion you are making it overly complicated. Since this is basically just the normal Chi-squared test for the goodness-of-fit with the null hypothesis that the observed frequency distribution is the same as the theoretical distribution. en.wikipedia.org/wiki/Pearson%27s_chi-squared_test#Definition Mar 20 '18 at 9:01
• @hannes101 This is not a type I error caused by large sample size. It is the large sample size detecting a real but tiny deviation from the null hypothesis.
– Dave
Jul 20 '20 at 16:11