Is it valid to rank goodness-of-fit based on the value of the Kolmogorov-Smirnov or Anderson-Darling test statistics?

Question

When using Kolmogorov-Smirnov or Anderson-Darling goodness-of-fit tests, is it valid to claim that, because distribution X has a lower test statistic than distribution Y, distribution X is a better fit than Y?

I believe that both KS and AD are go-no go tests, in that they can only say, given a significance level, whether it's reasonable to reject the hypothesis that the data comes from the distribution being tested for. As I understand them, these tests are inadequate for comparing the goodness-of-fit of two distributions if they are both deemed acceptable (i.e. if both of their test statistics are below a critical threshold).

In other words, I think the answer to my own question is no, I can't claim that X is a better fit for the data than Y just because the test statistic for X is lower than for Y. I can only tell whether I can reject the hypothesis that the data comes from each distribution X and Y.

However, a coworker disagreed with me, and I couldn't find any resources online to back up my opinion. Who is correct? If you could point me in the direction of said resources (proving me either right or wrong), that would be appreciated.

Cheers!

Answer 1

It depends on what you intend by "better fit".

Goodness of fit statistics measure deviation from perfect fit in some manner (we'll take it as given - as you assume in your question - that our statistic is organized such that smaller values mean closer fit to the distributional model in the null). Such statistics are sensitive to some kinds of deviation, and may be insensitive to other kinds of deviation.

If the kinds of deviation from the hypothesized model that the test statistic is good at picking up are the ones important for you to pick up (for whatever purpose you're testing for), then a smaller value of the statistic does indeed mean the fit is better (in the sense of 'better fit' defined by whatever you're trying to do).

If on the other hand there can be important deviations from the hypothesized model that the test statistic is not sensitive to then a smaller value doesn't necessarily mean better fit for your purposes.

[Note that since people's purposes may differ, what is a better fit for person 1 may not be a better fit for person 2.]

This makes it important to use a statistic that does pick up the sort of deviations that are important for you to pick up -- not just pick one at random. A well chosen statistic will then represent better or worse fit in a specific sense that's directly relevant to you.

I'm trying to figure out which distribution is best for a data set of pipe diameters. Since this amounts to the distribution of manufacturing deviations from spec (on top of measurement errors, of course), I believe the data might be normally distributed. There are also historical data which suggest that is usually the case with this particular data set. Do you think that KS is OK for this particular application?

No, I don't, for three reasons:

1. Nothing about your hypothesis specifies the parameter values for the normal distribution, and the Kolmogorov-Smirnov test is for a completely specified distribution.

2. While it's possible to use the same statistic for a general goodness of fit test for normality if you calculate new "tables" (a new distribution of the test statistic under H0), at which point you get Lilliefors' test, it's typically less powerful than the Shapiro-Wilk test, so you'd need good reason (such as a specific alternative in mind that it is better at) to choose the Lilliefors.

3. We actually can tell the underlying random variable isnt truly normal right from the start (pipe diameters are positive; manufacturing errors are also bounded, on at least one side). You must already know that before you even collect data. So the question "are these data drawn from a normal distribution?" is one you already know the answer to (no, they're not). Failure to reject would indicate that your sample size was too small to find it.Your actual question is more like "is the underlying distribution close enough to normal for my purposes?". That question isn't answered by any of these tests, it's nearer to a question about effect size (something more like how non-normal is the distribution and in what ways?).

[You might discern that many times that people test goodness of fit, they're not really addressing the question they need answered. One of many posts that have some discussion of issues like that is here.]

If you have a specific reason for wanting to check normality, that underlying reason may also tend to suggest ways to check its plausibility for that purpose.

So why do you need to know these values are from a normal distribution?

I'm trying to calculate a failure probability for this data set. So I have several variables (of which pipe diameter is one), a failure model (a formula that takes in these variables and outputs a resistance value) and some load cases, which allows me (in theory) to describe the joint distribution of the pipe resistance. In order to do that, I'd need to describe each of those variables' data sets. Hence the search for the distribution that best describes these data

No simple-form distribution is likely to 'best describe' your data -- real data are nearly always more complex than our simple models. To quote George Box --

Remember that all models are wrong; the practical question is how wrong do they have to be to not be useful
-- George Box & Norman R. Draper, Empirical Model-Building and Response Surfaces

(we can guess these are Box's words rather than Draper's because he's said very similar things elsewhere)

The relevant issue for you seems to be whether your model would be enable you to calculate failure probability accurately enough for your purposes. I see no reason why you'd use a goodness of fit test for any part of that, since a goodness of fit test simply doesn't address that issue.

---

(followup to discussion)

One way to assess the sensitivity to any assumed distribution for inputs in some Monte Carlo simulation of failure rates is to assume some other distribution is the real situation, simulate "real data" from that and see how much your assumption (including any fitting you're doing) affects the results of your subsequent Monte Carlo.

Answer 2

Your test gives you a bunch of statistics out, $(t_1, t_2, \dots, t_n)$, for each of the $n$ hypothesized distributions; you reject the hypothesis that the data are drawn from distribution $P_i$ at level $\alpha$ if $t_i > z_\alpha$.

Now your question is: is distribution $i$ a better fit than $j$ for some $t_i < t_j$? Glen_b addressed the problem of what "better" means, but let's assume you've drunk the Kool-Aid of whatever test you're running and believe that it's test statistic is the "right" measurement for your purposes. (You probably shouldn't drink this Kool-Aid; see Glen_b's answer.)

Note that, treating your observed data as random but the distributions as fixed, $t_i$ is random. So asking if $P_i$ is a better fit than $P_j$ could be thought of as asking something like: is $\DeclareMathOperator\E{\mathbb E} \E_{X \sim P^m} t_i(X) < \E_{X \sim P^m} t_j(X)$? Here by $P$ I mean the unknown true data distribution, and $P^m$ the distribution of iid samples from $P$ of size $m$. If $m$ is reasonably large and $t_i \ll t_j$, then we can say yes, $P_i$ is a better fit than $P_j$; if $m$ and/or the difference in statistics is small, we can't be sure that the difference isn't just random noise.

Actually working out the distribution of $t_i - t_j$, though, is difficult. The tests themselves only need to get the distribution of $t_i$ assuming $P_i = P$, but here we can't assume that. You could perhaps get a reasonable Monte Carlo guess at it by sampling from a density estimate, though.

Answer 3

I made some numerical experiments using random samples from a logistic distribution. Next I fit e.g. a normal model, a logistic model and a student-4 model. Depending on the observed random samples it can easily happen that the logistic fit is best according KS, but e.g. the normal model is better on likelihood L. Repeating such analysis many times we would observe that the logistic model is in average best according both L and KS. For large sample counts both L and KS become very stable and decisive, but L starts earlier to give stable decisions. For models with more differences in the distribution center the advantage of L is often smaller, e.g. if you want to differentiate a bimodal normal mix model and a triangular model. If your model application is more in the tail regions (e.g. high percentile estimation), then better trust L, because KS is not very sensitive to model deviations in that region (but AD is). If your data has outliers, then trusting blindly L is no good idea, whereas KS is quite a robust criteria. If you data is from a Quasi-MC analysis (e.g. LDS or LHS), then it can be smoother then pure rnd data. In such cases, GoF criteria like KS can become now more sensitive than L.