# Goodness of fit (cdf: empirical vs theoretical)?

I have a data-set with n = 90, probably follows the gamma distribution (and others). I used the maximum-likelihood estimation (MLE) to estimated the alpha and beta parameters of the gamma distribution using Matlab. What is the best way to test the fit (goodness of fit) of the gamma distribution with the estimated parameters versus the original data-set ?

Can I compare the cumulative distribution function (cdf) - empirical vs theoretical ?

empirical_cdf = ecdf ( data set )

theoretical_cdf = cdf ( gammafit )

And make same test, for example the KS two samples

kstest2 ( empirical_cdf, theoretical_cdf )

Is this the correct way ?

Many thanks

The histogram in the last question is only a example of 1 data-set (1 of 10000). I'll rephrase my question, I have a total of 10000 data-sets, and I wonder if the Gamma distribution is better (in terms of goodness-of-fit) that Weibull distribution for example.

or

For a data-set of 10000 what percentage fit better to gamma, and what percentage fit better to Weibull distribution ?

As you can see my data-set is big, and impossible to check one-by-one.

What is the best way to do the goodness-of-fit to found this percentages ?

Many thanks

• The statement "probably follows the gamma distribution (and others)" doesn't make sense to me. If it's gamma, it can't be anything that isn't gamma. If it's something that is not-gamma, it can't be gamma. A sample might reasonably be consistent with more than one distribution, but that doesn't mean it actually has any of the distributions you consider - and for certain it doesn't have more than one of them, if they're mutually exclusive. What is the reason you want to test goodness of fit? – Glen_b Oct 9 '14 at 1:08
• Increasing number of histogram bins (reducing of number of observations per bin) might suggest that the data should be modeled as a mixture of distributions rather than a single distribution (for example as a mixture of normals). Log transforming the data might also be useful to reveal patterns in the data. – Aleksandr Blekh Oct 9 '14 at 3:21

I don't use matlab, but how about we check the documentation of the function. It says:

kstest2

Kolmogorov-Smirnov test to compare the distribution of two samples

So no, that's not used for comparing a fitted distribution to a sample.

What about kstest? Well, if we check the documentation there, the answer is still no:

The Kolmogorov-Smirnov test requires that cdf be predetermined. It is not accurate if cdf is estimated from the data.

That pretty much covers it. There's a Lilliefors test (matlab has a function for the normal case, mentioned in the documentation for kstest). You could do something similar to that by simulating the distribution of the test statistic.

But often people test goodness of fit in situations in which it's not really useful to do so. (This may be the case here as well - why are you testing goodness of fit?)

• I'm curious about how inaccurate the K-S test is in ECDF case. I'd think it's still might make sense to perform the test to see the D- and p- values. It might be accurate enough for the OP's purposes. – Aleksandr Blekh Oct 9 '14 at 3:25
• @AleksandrBlekh Well, the D-value is fine in that it's still a meaningful quantity; its just that - because the distribution is fitted to the data, when the null is true, the fitted distribution is generally closer than the true distribution (if they were fitted by choosing the parameters to give the smallest KS distance - a perfectly reasonable approach - they'd always be smaller). ...(ctd) – Glen_b Oct 9 '14 at 5:09
• @AleksandrBlekh (ctd)... The distribution of D under the null is therefore substantially smaller than if you prespecify the distribution. Compare the Lilliefors table for the normal with the KS. For example, at n=20, $\alpha$=0.05, KS critical value is 0.294, while Lilliefors is 0.192 (~2/3 as big). The ratio of critical values is similar at n=50. – Glen_b Oct 9 '14 at 5:09
• @AleksandrBlekh If the performance at the two-parameter gamma is similar, when you look up KS tables but you should use Lilliefors, you will get wildly wrong p-values (the test will be highly conservative). Consider a difference that's right at the 5%CV for the Lilliefors at n=50 (testing normality in this case, since that's what I have tables for) ... if I've done it right, the KS will tell you the p-value is about 0.6 (!!) ... I think the effect gets smaller as n grows, so for very large $n$ it probably makes much less difference. – Glen_b Oct 9 '14 at 5:16
• Very interesting! I appreciate your comprehensive comments. Now I need some time to digest this information. – Aleksandr Blekh Oct 9 '14 at 6:04

Thank you for the clarification. This answer refers not to your original question, but to reformulated one. I would consider using a benchmarking package/library. I don't work with Matlab, so cannot suggest anything for that platform. However, if you can use R, I recommend you to take a look at the benchmark package: http://cran.r-project.org/web/packages/benchmark. Since you don't need to perform sampling (as your 10000 data sets are samples, if I understand correctly), you'd need to specify goodness-of-fit tests as algorithms in benchmark's terminology and run benchmarking across all data sets and all selected potential distributions (Gamma, Weibull, etc.). You could also use parametric distribution fitting packages/libraries prior to benchmarking to find the optimal parameters for the distributions. At least, this is the direction, which I would have explored first.