1
$\begingroup$

I have a basic question about the chi-squared statistic. Perhaps I am misunderstanding, or perhaps I am looking for a different test. I am mathematical to a degree but am (becoming) self-educated in statistics.

I understand how to calculate the statistic and do hypothesis testing, according to the textbooks. However, the statistic is very sensitive to the total frequency, in that, if X2(O,E) is the statistic, then increasing all the frequencies by some factor N increases the statistic by the same factor, i.e.,

X2(N*O,N*E)  = N * X2(O,E)          

where N*O is multiplying a vector by a scalar. Therefore, the statistic is much more sensitive to the total frequency than to the degrees of freedom. When testing goodness-of-fit for a distribution, it seems like the total frequency should NOT be the most significant factor.

This seems odd and I am wondering what I am missing, if anything.

$\endgroup$
1
  • 1
    $\begingroup$ Although the statistic changes with scaling, so does the reference distribution. Thus, looking at the statistic alone reveals nothing. $\endgroup$
    – whuber
    Oct 27, 2015 at 20:03

1 Answer 1

3
$\begingroup$

This is a general problem with goodness of fit tests.

If I fit a cancer risk model in 500,000 patients, I know I will have a nicely calibrated risk model. If I collect cancer outcomes in an independent set of 500,000 patients, I know that my previous model will fail most goodness of fit tests.

The reason for the problem is over-reliance on the $p$-value: the more $N$ you have, the more likely you are to reject the null hypothesis even if the differences are small and unassuming. Therefore, I look to the $p$-value moreover as a measure of sample size than anything. Of course, in your example under the null, "goodness-of-fit" is only achieved if $O = E$ exactly. In that case, the chi-square statistic will have an expected value of the degrees of freedom of the test, and the $p$-value is 0.5 every time.

However, you and I agree that small differences from $E$ and $O$ do not mean that the model fits poorly, and yet, if you were to sample ad infinitum, the test would ultimately reject and conclude that there is poor goodness of fit tests.

I summarily do not conduct goodness-of-fit tests without first visualizing the difference, if a picture cannot convince, neither should a statistical test.

$\endgroup$
3
  • $\begingroup$ Thanks, an excellent answer. So if we normalize all data to a probability distribution (bins sum to 1.0) and view them on the same graph (what I assume you mean by a picture), then what we are really doing is looking at the sum-of-absolute-values-of-differences. So perhaps this is a better statistical test of goodness-of-fit if we wish it to be independent of the population size? $\endgroup$
    – user93340
    Oct 27, 2015 at 19:58
  • $\begingroup$ @user93340 There is no such test of which I'm aware. I have seen a few references, e.g. Cummings Rivara 2003 that suggest that there's really no reason to report such tests in papers. Therefore, you should use whatever method you feel is appropriate for assessing the goodness-of-fit, and if the collaborators, reviewers, or whoever is concerned with the issue, you can simply give them a gentle walkthrough. $\endgroup$
    – AdamO
    Oct 27, 2015 at 23:11
  • $\begingroup$ @user93340 for ad-hoc testing, a rule of thumb that I use often is to set significance levels for such tests based on a $p$-value calculated as: $p = 5/n$ with the thought that $p < 0.05$ is only appropriate for datasets of size 100 or so. $\endgroup$
    – AdamO
    Oct 27, 2015 at 23:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.