6
$\begingroup$

I'm working on samples that I'm trying to fit into log-normal distributions. In some cases, Kolmogorov-Smirnov test statistics is something like D = 0.0056 with an associated p-value of 0. Hence, my sample shows very small departures from a theoretical log-normal distribution, but looking at the p-value I reject the null hypothesis that that my sample is drawn from the reference distribution (log-normal).

KS-test is performed through R code:

sample.z <- std(sample) # I standardize data to allow for comparisons
LN <- rlnorm(1e5, 0, 1) # theoretical lognormal with mean = 0 and sigma = 1
ks.test(sample.z, LN)

Looking at the QQ-plot I see significant departures in the tails of my distributions. Thus, I was thinking that maybe I'm getting these results because of Pareto tails in my approximately log-normal distributions. Indeed, 

library(igraph)
power.law.fit(sample)

confirms this hypothesis by identifying a Pareto tail after a certain lower bound (xmin) with a certain alpha.

Now, I would like to show the fit of my Pareto tail in the QQ-plot. How can I do that? Can you suggest other data-visualization methods to stress out the presence of Pareto tails in log-normal distributions?

PDFs

CDFs fitdist output

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ "but looking at the p-value I fail to reject the alternative hypothesis that that my sample is log-normally distributed" -- this doesn't make sense. For starters, wasn't your null that it was lognormal? If that's so, how can the alternative also be that it was lognormal?? $\endgroup$
    – Glen_b
    Apr 21, 2014 at 9:16
  • $\begingroup$ Ops, I screwed up with the explanation. The null hypothesis is: samples are drawn from the reference distribution. The alternative hypothesis is: samples are NOT drawn from the reference distribution. $\endgroup$
    – stochazesthai
    Apr 21, 2014 at 9:22
  • $\begingroup$ Second, you don't reject or fail to reject the alternative; that only applies to the null. $\endgroup$
    – Glen_b
    Apr 21, 2014 at 9:23
  • 2
    $\begingroup$ I usually find P-values of little help here. Often the sample is so big that any deviation is flagged as significant at conventional levels. The key question is which of various candidate distributions fits best. We don't get a sense from your report (no data, no graphs) of whether you are talking about primary or secondary features. If the lognormal is approximately right, then your distribution is unimodal, and the Pareto is qualitatively wrong. That doesn't stop your distribution having e.g. heavier tails than predicted by the lognormal if that's what you are getting at. $\endgroup$
    – Nick Cox
    Apr 21, 2014 at 9:35
  • $\begingroup$ Thank you so much Nick. Now I will edit the post and I will put in some graphs. $\endgroup$
    – stochazesthai
    Apr 21, 2014 at 9:41

3 Answers 3

Reset to default
4
$\begingroup$

If you take logs, it should be normal with exponential tail

Just do a normal and an exponential qq plot of the data, the first should be roughly linear before the kink, the second roughly linear after the kink:

enter image description here

(In this case the change point was at 5.5, and we see what we should - a kink near 5.5, and the first plot roughly linear before and the second roughly linear after the kink. The fact that the first plot looks roughly linear after the kink as well suggests that the Pareto data might in this particular example have been reasonably approximated by a second lognormal.)

Improve this answer
$\endgroup$
1
$\begingroup$

Here we go. Normal QQ plot and Exponential QQ plot.

Normal Q-Q plot

Exponential Q-Q plot

Improve this answer
$\endgroup$
3
  • $\begingroup$ This isn't an answer, but an extension of your question. If you're seeking R-specific help, that's off-topic here. $\endgroup$
    – Nick Cox
    Apr 21, 2014 at 10:23
  • $\begingroup$ Ok, I'll ask for help somewhere and I'll come back with the plot. Thanks. $\endgroup$
    – stochazesthai
    Apr 21, 2014 at 10:31
  • 1
    $\begingroup$ It certainly seems consistent with your thesis that the original (untransformed) data are close to lognormal in the body of the distribution but with an approximately Pareto tail. These images could perhaps be added to the end of your question as a followup. $\endgroup$
    – Glen_b
    Apr 21, 2014 at 16:19
0
$\begingroup$

If you are interested in testing the Pareto tail, this answer would help you. If you are interested in visualizing the Pareto tail, this gist can plot the empirical CCDF of your data in log scale. A Pareto tail would manifest itself in a straight line.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.