17
$\begingroup$

I'm trying to determine if my dataset of continuous data follows a gamma distribution with parameters shape $=$ 1.7 and rate $=$ 0.000063.

The problem is when I use R to create a Q-Q plot of my dataset $x$ against the theoretical distribution gamma (1.7, 0.000063), I get a plot that shows that the empirical data roughly agrees with the gamma distribution. The same thing happens with the ECDF plot.

However when I run a Kolmogorov-Smirnov test, it gives me an unreasonably small $p$-value of $<1\%$.

Which should I choose to believe? The graphical output or the result from KS-test?

QQplot and ECDF plot

Improve this question
$\endgroup$
2
  • $\begingroup$ can you also provide the density distribution plots you obtain ? $\endgroup$
    – Scratch
    Commented Jan 17, 2014 at 11:34
  • 13
    $\begingroup$ The test and the diagnostic plot aren't inconsistent. The distribution is similar to the theoretical one, as the QQ plot shows. The sample size is large enough that you are likely to pick up even small differences from the theoretical one. $\endgroup$
    – Glen_b
    Commented Jan 17, 2014 at 13:21

4 Answers 4

Reset to default
20
$\begingroup$

I don't see any sense in not "believing" the Q-Q plot (if you've produced it properly); it's just a graphical representation of the reality of your data, juxtaposed with the definitional distribution. Clearly it's not a perfect match, but if it's good enough for your purposes, that may be more or less the end of the story. You may want to check out this related question: Is normality testing 'essentially useless'?

The $p$-value from the KS test is basically telling you that your sample size is large enough to give strong evidence against the null hypothesis that your data belong to exactly the same distribution as your reference distribution (I assume you referenced the gamma distribution; you may want to double-check that you did). That seems clear enough from the Q-Q plot as well (i.e., there are some small but seemingly systematic patterns of deviation), so I don't think there's truly any conflicting information here.

Whether your data are too different from a gamma distribution for your intended purposes is another question. The KS test alone can't answer it for you (because its outcome will depend on your sample size, among other reasons), but the Q-Q plot might help you decide. You might also want to look into robust alternatives to any other analyses you plan to run, and if you're particularly serious about minding the sensitivity of any subsequent analyses to deviations from the gamma distribution, you might want to consider doing some simulation testing too.

Improve this answer
$\endgroup$
17
$\begingroup$

What you could do is create multiple samples from your theoretical distribution and plot those on the background of your QQ-plot. That will give you an idea of what kind of variability you can reasonably expect from just sampling.

You can extend that idea to create an envelope around the theoretical line, using the example from pages 86-89 of :

Venables, W.N. and Ripley, B.D. 2002. Modern applied statistics with S. New York: Springer.

This will be a point-wise envelope. You can extend that idea even further to create an overall envelope using the ideas from pages 151-154 of:

Davison, A.C. and Hinkley, D.V. 1997. Bootstrap methods and their application. Cambridge: Cambridge University Press.

However, for basic exploration I think just plotting a couple of reference samples in the background of your QQ-plot will be more than enough.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Good idea! Remind me to upvote this in 11 hours (used up all my votes on cartoons)...I particularly like bootstrapping the ECDF as a way of enriching that kind of plot. $\endgroup$
    – Nick Stauner
    Commented Jan 17, 2014 at 12:39
  • 1
    $\begingroup$ Also have a look at CRAN package sfsmisc, which has the function ecdf.ksCI draweing a confidence band on the ecdf plot. The same idea could be used to draw a confidence band on the QQ plot ... $\endgroup$
    – kjetil b halvorsen ♦
    Commented Apr 8, 2015 at 12:41
2
$\begingroup$

The KS test assumes particular parameters of your distribution. It tests the hypothesis "the data are distributed according to this particular distribution". You might have specified these parameters somewhere. If not, some not matching defaults may have been used. Note that the KS test will become conservative if the estimated parameters are plugged into the hypothesis.

However, most goodness-of-fit tests are used the wrong way round. If the KS test would not have shown significance, this does not mean that the model you wanted to prove is appropriate. That's what @Nick Stauner said about too small sample size. This issue is similar to point hypothesis tests and equivalence tests.

So in the end: Only consider the QQ-plots.

Improve this answer
$\endgroup$
-1
$\begingroup$

QQ Plot is an exploratory data analysis technique and should be treated as such - so are all other EDA plots. They are only meant to give you preliminary insights into the data on hand. You should never decide or stop your analysis based on EDA plots like QQ plot. It is a wrong advice to consider only QQ plots. You should definitely go by quantiative techniques like KS Test. Suppose you have another QQ plot for similar data set, how would you compare the two without a quantitative tool? Right next step for you, after EDA and KS test is to find out why KS test is giving low p-value (in your case, it could even be due to some error).

EDA techniques are NOT meant to serve as decision making tools. In fact, I would say even inferential statistics are meant to be only exploratory. They give you pointers as to which direction your statistical analysis should proceed. For example, a t-test on a sample would only give you a confidence level that the sample may (or may not) belong to the population, you may still proceed further based on that insight as to what distribution your data belongs to and what are its parameters etc. In fact, when some state that even techniques implemented as part of machine learning libraries are also exploratory in nature!!! I hope they mean it in this sense...!

To conclude statistical decisions on the basis of plots or visualization techniques is making mockery of advances made in statistical science. If you ask me, you should use these plots as tools for communicating the final conclusions based on your quantitative statistical analysis.

Improve this answer
$\endgroup$
7
  • $\begingroup$ This forbids me from doing something that I do often and regard as sensible, make a decision given an exploratory plot and stop before a more formal significance test. No mockery is entailed. This is a repetitive and dogmatic comment that doesn't add anything useful to existing excellent, and much more nuanced, answers. It's very easy to compare QQ plots... $\endgroup$
    – Nick Cox
    Commented Feb 11, 2019 at 9:05
  • $\begingroup$ I have not read other answers but if they also encourage quantitative methods, I am fine. For the question asked, I had given my answer. But, I am curious, it does not take much time to do formal quant tests (just few more minutes to do KS test) with now available packages like R, so why would anybody stop at EDA plots? Just after validating KS test results of R with bootstrapping, I noticed in several places where it was mentioned as not preferable to use etc.,.. Is it due to a general suspicion about traditional stat methods? This is the rationale behind my strong comments..not to offend any $\endgroup$
    – Murugesan Narayanaswamy
    Commented Feb 11, 2019 at 9:58
  • $\begingroup$ You really should read other answers before posting. The implication of posting is that you have something different (as well as defensible) to say. Your comment is puzzling in implying that Q-Q plots are not "quantitative methods". A Q-Q plot shows in principle all the quantitative information relevant in assessing distribution fit. In contrast a test like Kolmogorov-Smirnov gives a one-dimensional reduction and gives little help on what to do next. $\endgroup$
    – Nick Cox
    Commented Feb 11, 2019 at 10:23
  • $\begingroup$ Q-Q plot compares theoretical distribution with given test data and provides a visual representation but KS test does the same thing in much more rigorous way using statistical concepts and gives finally a probability value. You cannot compare two QQ plots but you will get a quantiative difference when you use KS test. It is misnomer that KS test p-value is wrong. It is also wrong that empirical data set cannot be used to extract distribution parameters. I have personally done bootstrapping and verified with p values with both tables and manually calculated kolomogrov distribution. $\endgroup$
    – Murugesan Narayanaswamy
    Commented Feb 11, 2019 at 12:52
  • $\begingroup$ There is much shadow boxing in your comment, Who is arguing where that you can't use empirical data to get parameter estimates? That is what we should all agree is being done here. You'll have to forgive me for not wanting to pursue a discussion. I stand by my reaction to your answer. $\endgroup$
    – Nick Cox
    Commented Feb 11, 2019 at 13:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.