# Conflicting results with GoF tests

I am a computer scientist with little statistics background and I am trying to find the best fitting distribution for some data set (using MATLAB). To assess the goodness of fit I use both Kolmogorov Smirnov (KS) and Anderson Darling (AD) tests, and here are the p-values for the same data set:

Distribution          AD Test     KS Test
Exp                   0.439       1.49e-7
Weibull               0.498       1.40e-6
Pareto                0.244       6.24e-14
Logn                  0.684       2.69e-4
Gamma                 0.595       2.16e-4


I use a significance level of 0.05, and as far as I know with a p-value < 0.05 the null hypothesis is rejected which is the case for the KS test results. Then what should I conclude based on this result? The KS test says that none of the distributions is a good fit while the AD test can't reject the null hypothesis.

Edit: Here is an overview of what we do in our code:

fit_functions = { @wblfit, @expfit, @lognfit, @gpfit, @gamfit};
for i=1:length(fit_functions)
[varargout{1:x}] = fit (param, fit_functions {i});
[ad_result ks_result] = run_gof_tests(param, cdf_functions{i}, ... varargout{:});


and in run_gof_tests function we average 1,000 p-values to calculate the final p-values. Each p-value is computed by drawing fifty samples randomly from the data set. We have used this method due to reasons described in this tech report on pg 12: Modeling Machine Availability in Enterprise and Wide-area Distributed Computing Environments by Nurmi et al., UCSB Computer Science Technical Report Number CS2003-28.

Thanks!

• Certainly looks odd for the results to be so discrepant. Might help if you post your code so those who speak MATLAB (not me) can check it. Jan 9, 2012 at 13:37
• I agree with @onestop: these two tests are supposed to be evaluating the same thing; if so, their p-values cannot be orders of magnitude off. Also, I would be interested in learning the reasons described in that tech report, because my initial reaction to this approach was "that's silly" and upon further reflection I can see that it makes a little sense to average p-values from testing subsamples, but I'm baffled as to why one would elect to lose so much information in conducting a test. In other words, it might not be silly, but it sure looks inferior.
– whuber
Jan 9, 2012 at 15:01
• I note you say you're averaging p-values, while the tech report you reference (p12-13) says "we use the average test statistic value to compute the p-value". The two procedures are certainly not equivalent. Jan 9, 2012 at 15:34
• But they also say on p13 that "Test results are shown in Table 2 which are the average p-values from the 1000 iterations of the test". So they are talking about both "the p-value" as you point out and "the average p-values" which confuses me. Jan 9, 2012 at 15:46
• I find the method described in that report to be highly suspect. It makes no sense to base a p-value on averages of test statistics from subsamples (a single outlying subsample could really screw things up). It makes a little sense to average the p-values of those statistics, but the purpose is bizarre: namely, to reduce the power of the test! There have to be much better, theoretically justifiable ways to accomplish the same thing. One should begin with a single test statistic rather than simulating it from subsamples.
– whuber
Jan 9, 2012 at 19:49