# Which to believe: Kolmogorov-Smirnov test or Q-Q plot?

I'm trying to determine if my dataset of continuous data follows a gamma distribution with parameters shape $=$ 1.7 and rate $=$ 0.000063.

The problem is when I use R to create a Q-Q plot of my dataset $x$ against the theoretical distribution gamma (1.7, 0.000063), I get a plot that shows that the empirical data roughly agrees with the gamma distribution. The same thing happens with the ECDF plot.

However when I run a Kolmogorov-Smirnov test, it gives me an unreasonably small $p$-value of $<1\%$.

Which should I choose to believe? The graphical output or the result from KS-test?

-
can you also provide the density distribution plots you obtain ? –  Scratch Jan 17 at 11:34
The test and the diagnostic plot aren't inconsistent. The distribution is similar to the theoretical one, as the QQ plot shows. The sample size is large enough that you are likely to pick up even small differences from the theoretical one. –  Glen_b Jan 17 at 13:21

## 3 Answers

I don't see any sense in not "believing" the Q-Q plot (if you've produced it properly); it's just a graphical representation of the reality of your data, juxtaposed with the definitional distribution. Clearly it's not a perfect match, but if it's good enough for your purposes, that may be more or less the end of the story. You may want to check out this related question: Is normality testing 'essentially useless'?

The $p$-value from the KS test is basically telling you that your sample size is large enough to give strong evidence against the null hypothesis that your data belong to exactly the same distribution as your reference distribution (I assume you referenced the gamma distribution; you may want to double-check that you did). That seems clear enough from the Q-Q plot as well (i.e., there are some small but seemingly systematic patterns of deviation), so I don't think there's truly any conflicting information here.

Whether your data are too different from a gamma distribution for your intended purposes is another question. The KS test alone can't answer it for you (because its outcome will depend on your sample size, among other reasons), but the Q-Q plot might help you decide. You might also want to look into robust alternatives to any other analyses you plan to run, and if you're particularly serious about minding the sensitivity of any subsequent analyses to deviations from the gamma distribution, you might want to consider doing some simulation testing too.

-

What you could do is create multiple samples from your theoretical distribution and plot those on the background of your QQ-plot. That will give you an idea of what kind of variability you can reasonably expect from just sampling.

You can extend that idea to create an envelope around the theoretical line, using the example from pages 86-89 of :

Venables, W.N. and Ripley, B.D. 2002. Modern applied statistics with S. New York: Springer.

This will be a point-wise envelope. You can extend that idea even further to create an overall envelope using the ideas from pages 151-154 of:

Davison, A.C. and Hinkley, D.V. 1997. Bootstrap methods and their application. Cambridge: Cambridge University Press.

However, for basic exploration I think just plotting a couple of reference samples in the background of your QQ-plot will be more than enough.

-
Good idea! Remind me to upvote this in 11 hours (used up all my votes on cartoons)...I particularly like bootstrapping the ECDF as a way of enriching that kind of plot. –  Nick Stauner Jan 17 at 12:39

The KS test assumes particular parameters of your distribution. It tests the hypothesis "the data are distributed according to this particular distribution". You might have specified these parameters somewhere. If not, some not matching defaults may have been used. Note that the KS test will become conservative if the estimated parameters are plugged into the hypothesis.

However, most goodness-of-fit tests are used the wrong way round. If the KS test would not have shown significance, this does not mean that the model you wanted to prove is appropriate. That's what @Nick Stauner said about too small sample size. This issue is similar to point hypothesis tests and equivalence tests.

So in the end: Only consider the QQ-plots.

-