Which to believe: Kolmogorov-Smirnov test or Q-Q plot?

Question

I'm trying to determine if my dataset of continuous data follows a gamma distribution with parameters shape $=$ 1.7 and rate $=$ 0.000063.

The problem is when I use R to create a Q-Q plot of my dataset $x$ against the theoretical distribution gamma (1.7, 0.000063), I get a plot that shows that the empirical data roughly agrees with the gamma distribution. The same thing happens with the ECDF plot.

However when I run a Kolmogorov-Smirnov test, it gives me an unreasonably small $p$-value of $<1\%$.

Which should I choose to believe? The graphical output or the result from KS-test?

Answer 1

I don't see any sense in not "believing" the Q-Q plot (if you've produced it properly); it's just a graphical representation of the reality of your data, juxtaposed with the definitional distribution. Clearly it's not a perfect match, but if it's good enough for your purposes, that may be more or less the end of the story. You may want to check out this related question: Is normality testing 'essentially useless'?

The $p$-value from the KS test is basically telling you that your sample size is large enough to give strong evidence against the null hypothesis that your data belong to exactly the same distribution as your reference distribution (I assume you referenced the gamma distribution; you may want to double-check that you did). That seems clear enough from the Q-Q plot as well (i.e., there are some small but seemingly systematic patterns of deviation), so I don't think there's truly any conflicting information here.

Whether your data are too different from a gamma distribution for your intended purposes is another question. The KS test alone can't answer it for you (because its outcome will depend on your sample size, among other reasons), but the Q-Q plot might help you decide. You might also want to look into robust alternatives to any other analyses you plan to run, and if you're particularly serious about minding the sensitivity of any subsequent analyses to deviations from the gamma distribution, you might want to consider doing some simulation testing too.

Answer 2

What you could do is create multiple samples from your theoretical distribution and plot those on the background of your QQ-plot. That will give you an idea of what kind of variability you can reasonably expect from just sampling.

You can extend that idea to create an envelope around the theoretical line, using the example from pages 86-89 of :

Venables, W.N. and Ripley, B.D. 2002. Modern applied statistics with S. New York: Springer.

This will be a point-wise envelope. You can extend that idea even further to create an overall envelope using the ideas from pages 151-154 of:

Davison, A.C. and Hinkley, D.V. 1997. Bootstrap methods and their application. Cambridge: Cambridge University Press.

However, for basic exploration I think just plotting a couple of reference samples in the background of your QQ-plot will be more than enough.

Answer 3

The KS test assumes particular parameters of your distribution. It tests the hypothesis "the data are distributed according to this particular distribution". You might have specified these parameters somewhere. If not, some not matching defaults may have been used. Note that the KS test will become conservative if the estimated parameters are plugged into the hypothesis.

However, most goodness-of-fit tests are used the wrong way round. If the KS test would not have shown significance, this does not mean that the model you wanted to prove is appropriate. That's what @Nick Stauner said about too small sample size. This issue is similar to point hypothesis tests and equivalence tests.

So in the end: Only consider the QQ-plots.

Answer 4

QQ Plot is an exploratory data analysis technique and should be treated as such - so are all other EDA plots. They are only meant to give you preliminary insights into the data on hand. You should never decide or stop your analysis based on EDA plots like QQ plot. It is a wrong advice to consider only QQ plots. You should definitely go by quantiative techniques like KS Test. Suppose you have another QQ plot for similar data set, how would you compare the two without a quantitative tool? Right next step for you, after EDA and KS test is to find out why KS test is giving low p-value (in your case, it could even be due to some error).

EDA techniques are NOT meant to serve as decision making tools. In fact, I would say even inferential statistics are meant to be only exploratory. They give you pointers as to which direction your statistical analysis should proceed. For example, a t-test on a sample would only give you a confidence level that the sample may (or may not) belong to the population, you may still proceed further based on that insight as to what distribution your data belongs to and what are its parameters etc. In fact, when some state that even techniques implemented as part of machine learning libraries are also exploratory in nature!!! I hope they mean it in this sense...!

To conclude statistical decisions on the basis of plots or visualization techniques is making mockery of advances made in statistical science. If you ask me, you should use these plots as tools for communicating the final conclusions based on your quantitative statistical analysis.