2
$\begingroup$

Below is the histogram and the model distribution which goodness-of fit that I want to test

enter image description here

Judging by the eye, I would say this is a pretty good fit.

However, when I run the chi square test, it fails. The data for observed count, and expected count is as follows:

0   903 1730    2637
2937    3678    3539    3870
3353    3406    2810    2843
2098    2051    1577    1426
982 873 601 521
385 342 232 197
127 104 57  52
23  16  17  10
7   6       

0.00    847.25  1707.92     2512.40 
3131.78     3484.96     3600.03     3565.79 
3449.83     3272.62     3033.35     2737.29 
2402.19     2051.91     1709.04     1390.81 
1107.99     865.54  663.95  500.70 
371.55  271.49  195.43  138.64 
96.95   66.83   45.41   30.42 
20.08   13.06   8.36    5.27 
3.28    2.00

By calculation, chi square statistics is 284. On the loosest standard, a chi square(34) with 1 - a = 0.95 is 48.6

Since 284 > 48.6, the distribution should be rejected.

Really? Or Did I made any mistake?

I take a rough calculation, for 48.6/34 = 1.3, which is the average of accepting level of variance. However, take an example for the 3rd bin, (2637-2512)^2 = 15625, which is 6 times of the expected frequency, 2512. Is it too strict?

Possible useful links:

sample size too large?

Are large data sets inappropriate for hypothesis testing?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ My eye says the fit and the data are clearly different. And therein lies the utility of formal hypothesis tests: they help remove the subjective, undocumentable, and irreproducible differences that can arise among multiple opinions. If the test is appropriate for the data, then the result is indisputable. What you should instead be asking is whether the size of the difference it detects is of any concern. $\endgroup$
    – whuber ♦
    Feb 2, 2016 at 14:36
  • $\begingroup$ @whuber, in my case, I think the difference between the model and the histogram is acceptable (I understand that the data retrive process would introduce a lot of variance like this). And It's totally OK to be rejected for some gof test. The problem is that, there is some fit here, and I don't know if there is other good way of measuring it and expressing it. I have R square, but it's too insensitve, while chi square is too sensitive. $\endgroup$
    – ZK Zhao
    Feb 2, 2016 at 15:29
  • 1
    $\begingroup$ Sensitivity is a good thing when it comes to measuring fits. Regardless, how you measure the goodness of fit ought to depend on what you're attempting to accomplish. As an example, if it is crucial in an application to model the largest values accurately, then the goodness of fit measurement ought to weight the upper tail of the distribution more heavily than the rest. I hope it's clear that these considerations are independent of, and separate from, whether the differences may be statistically "significant." $\endgroup$
    – whuber ♦
    Feb 2, 2016 at 16:27

1 Answer 1

Reset to default
6
$\begingroup$

It is a well-known fact that frequentist hypothesis tests can reject the null hypothesis due to differences that are of extremely small size, if the sample size is large enough. In your case the match of the data with the theoretical distribution indeed seems reasonably good, but your sample size is also extremely large. In some cases such a rejection of the null hypothesis may be irrelevant (e.g. if there is only a small deviation of residuals from normality for a linear regression model) and sometimes this is what one is looking for.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Then is there any other approach that I can turn to? I may try K-S test instead. Or any approach that I can quantify the certainty of goodness? Chi square may be too strict for my case, and I don't know how can I make it looser. $\endgroup$
    – ZK Zhao
    Feb 2, 2016 at 6:56
  • 5
    $\begingroup$ @cqcn1991, you need to distinguish between statistical significance and subject-matter significance. See this excellent thread. $\chi^2$ test is fine in what it addresses (statistical significance) and I suspect K-S should not differ from it too much. $\endgroup$
    – Richard Hardy
    Feb 2, 2016 at 7:52
  • 2
    $\begingroup$ @cqcn1991 The chi-square is one of the lowest power goodness of fit tests in common use. Any decent test of goodness of fit would reject much more often. The KS is not high power but it leaves the chi-square in the dust. $\endgroup$
    – Glen_b
    Feb 2, 2016 at 7:58
  • 1
    $\begingroup$ The alternative is what you have already done, plot and see if it fits. If you need a fancy name for it, you can call it the interoccular trauma test (does it hit you between the eyes test). $\endgroup$
    – Maarten Buis
    Feb 2, 2016 at 12:40
  • 3
    $\begingroup$ With an indicator you try to quantify the difference between your theoretical and observed data. Both R2 and all the goodness of fit tests do that. The problem is which cut-off-value do you choose. Statistical tests have very particular way of choosing the cut-off point, which is typically not what you want in large datasets. They are not wrong, they just answer a different questions than the one you are interested in. The interoccular trauma test is a good alternative. However, if you want to automate that you need to find some other value, which will always be arbitrary. $\endgroup$
    – Maarten Buis
    Feb 2, 2016 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.