# Conflicting results with GoF tests

I am a computer scientist with little statistics background and I am trying to find the best fitting distribution for some data set (using MATLAB). To assess the goodness of fit I use both Kolmogorov Smirnov (KS) and Anderson Darling (AD) tests, and here are the p-values for the same data set:

Distribution          AD Test     KS Test
Exp                   0.439       1.49e-7
Weibull               0.498       1.40e-6
Pareto                0.244       6.24e-14
Logn                  0.684       2.69e-4
Gamma                 0.595       2.16e-4


I use a significance level of 0.05, and as far as I know with a p-value < 0.05 the null hypothesis is rejected which is the case for the KS test results. Then what should I conclude based on this result? The KS test says that none of the distributions is a good fit while the AD test can't reject the null hypothesis.

Edit: Here is an overview of what we do in our code:

fit_functions = { @wblfit, @expfit, @lognfit, @gpfit, @gamfit};
for i=1:length(fit_functions)
[varargout{1:x}] = fit (param, fit_functions {i});
[ad_result ks_result] = run_gof_tests(param, cdf_functions{i}, ... varargout{:});


and in run_gof_tests function we average 1,000 p-values to calculate the final p-values. Each p-value is computed by drawing fifty samples randomly from the data set. We have used this method due to reasons described in this tech report on pg 12: Modeling Machine Availability in Enterprise and Wide-area Distributed Computing Environments by Nurmi et al., UCSB Computer Science Technical Report Number CS2003-28.

Thanks!

• Certainly looks odd for the results to be so discrepant. Might help if you post your code so those who speak MATLAB (not me) can check it. Jan 9 '12 at 13:37
• I agree with @onestop: these two tests are supposed to be evaluating the same thing; if so, their p-values cannot be orders of magnitude off. Also, I would be interested in learning the reasons described in that tech report, because my initial reaction to this approach was "that's silly" and upon further reflection I can see that it makes a little sense to average p-values from testing subsamples, but I'm baffled as to why one would elect to lose so much information in conducting a test. In other words, it might not be silly, but it sure looks inferior.
– whuber
Jan 9 '12 at 15:01
• I note you say you're averaging p-values, while the tech report you reference (p12-13) says "we use the average test statistic value to compute the p-value". The two procedures are certainly not equivalent. Jan 9 '12 at 15:34
• But they also say on p13 that "Test results are shown in Table 2 which are the average p-values from the 1000 iterations of the test". So they are talking about both "the p-value" as you point out and "the average p-values" which confuses me. Jan 9 '12 at 15:46
• I find the method described in that report to be highly suspect. It makes no sense to base a p-value on averages of test statistics from subsamples (a single outlying subsample could really screw things up). It makes a little sense to average the p-values of those statistics, but the purpose is bizarre: namely, to reduce the power of the test! There have to be much better, theoretically justifiable ways to accomplish the same thing. One should begin with a single test statistic rather than simulating it from subsamples.
– whuber
Jan 9 '12 at 19:49