Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm working on samples that I'm trying to fit into log-normal distributions. In some cases, Kolmogorov-Smirnov test statistics is something like D = 0.0056 with an associated p-value of 0. Hence, my sample shows very small departures from a theoretical log-normal distribution, but looking at the p-value I reject the null hypothesis that that my sample is drawn from the reference distribution (log-normal).

KS-test is performed through R code:

sample.z <- std(sample) # I standardize data to allow for comparisons
LN <- rlnorm(1e5, 0, 1) # theoretical lognormal with mean = 0 and sigma = 1
ks.test(sample.z, LN)

Looking at the QQ-plot I see significant departures in the tails of my distributions. Thus, I was thinking that maybe I'm getting these results because of Pareto tails in my approximately log-normal distributions. Indeed,

library(igraph)
power.law.fit(sample)

confirms this hypothesis by identifying a Pareto tail after a certain lower bound (xmin) with a certain alpha.

Now, I would like to show the fit of my Pareto tail in the QQ-plot. How can I do that? Can you suggest other data-visualization methods to stress out the presence of Pareto tails in log-normal distributions?

PDFs

CDFs fitdist output

share|improve this question
1  
"but looking at the p-value I fail to reject the alternative hypothesis that that my sample is log-normally distributed" -- this doesn't make sense. For starters, wasn't your null that it was lognormal? If that's so, how can the alternative also be that it was lognormal?? –  Glen_b Apr 21 at 9:16
    
Ops, I screwed up with the explanation. The null hypothesis is: samples are drawn from the reference distribution. The alternative hypothesis is: samples are NOT drawn from the reference distribution. –  stochazesthai Apr 21 at 9:22
    
Second, you don't reject or fail to reject the alternative; that only applies to the null. –  Glen_b Apr 21 at 9:23
2  
I usually find P-values of little help here. Often the sample is so big that any deviation is flagged as significant at conventional levels. The key question is which of various candidate distributions fits best. We don't get a sense from your report (no data, no graphs) of whether you are talking about primary or secondary features. If the lognormal is approximately right, then your distribution is unimodal, and the Pareto is qualitatively wrong. That doesn't stop your distribution having e.g. heavier tails than predicted by the lognormal if that's what you are getting at. –  Nick Cox Apr 21 at 9:35
    
Thank you so much Nick. Now I will edit the post and I will put in some graphs. –  stochazesthai Apr 21 at 9:41

2 Answers 2

If you take logs, it should be normal with exponential tail

Just do a normal and an exponential qq plot of the data, the first should be roughly linear before the kink, the second roughly linear after the kink:

enter image description here

(In this case the change point was at 5.5, and we see what we should - a kink near 5.5, and the first plot roughly linear before and the second roughly linear after the kink. The fact that the first plot looks roughly linear after the kink as well suggests that the Pareto data might in this particular example have been reasonably approximated by a second lognormal.)

share|improve this answer

Here we go. Normal QQ plot and Exponential QQ plot.

Normal Q-Q plot

Exponential Q-Q plot

share|improve this answer
    
This isn't an answer, but an extension of your question. If you're seeking R-specific help, that's off-topic here. –  Nick Cox Apr 21 at 10:23
    
Ok, I'll ask for help somewhere and I'll come back with the plot. Thanks. –  stochazesthai Apr 21 at 10:31
1  
It certainly seems consistent with your thesis that the original (untransformed) data are close to lognormal in the body of the distribution but with an approximately Pareto tail. These images could perhaps be added to the end of your question as a followup. –  Glen_b Apr 21 at 16:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.