11
$\begingroup$

I'd like to check in R if my data fits log-normal or Pareto distributions. How could I do that? Perhaps ks.test could help me do that, but how could I get the $\alpha$ and $k$ parameters for Pareto distribution for my data?

$\endgroup$
6
$\begingroup$

... I've just noticed you have the 'regression' tag there. If you do have a regression problem you can't look at the univariate distribution of the response to assess the distributional shape, since it depends on the pattern of the x's. If you're asking about checking whether a response (y) variable in some kind of regression or GLM has a lognormal or a Pareto distribution where the means differ across observation, that's a very different question (but basically comes down to similar kinds of analysis on the residuals). Can you please clarify if it is a regression problem. My answer, at present, relates to assessing univariate lognormal or Pareto

You have some quite different questions there.

How to check if my data fits log normal distribution?

Take logs and do a normal QQ plot. Look and see if the distribution is close enough for your purposes.

I'd like to check in R if my data fits log-normal or Pareto distributions

Accept from the start that none of the distributions you consider will be am exact description. You're looking for a reasonable model. This means that at small sample sizes, you won't reject any reasonable option, but with sufficient sample size you'll reject them all. Worse, with large sample size, you'll reject perfectly decent models, while at small sample sizes you won't reject bad ones.

Such tests aren't really a useful basis for model selection.

In short, your question of interest - something like "what's a good model for this data, one that is close enough that it will make subsequent inference useful?" is simply not answered by goodness of fit tests. However, in some cases goodness of fit statistics (rather than decisions coming out of rejection rules based on them) may in some cases provide a useful summary of particular kinds of lack of fit.

Perhaps ks.test could help me do that

No. First, there's the issue I just mentioned, and second, a Kolmogorov-Smirnov test is a test for a completely specified distribution. You don't have one of those.

In many cases, I'd recommend Q-Q plots and similar displays. For right skew cases like this, I'd tend to work with logs (a lognormal will then look normal, while a Pareto will look exponential). At reasonable sample sizes it's not hard to distinguish visually whether data looks more nearly normal than exponential or vice versa. First, get some actual data from each and plot those - say half a dozen samples at least, so you know what they look like.

See an example below

how could I get the alpha and k parameters for pareto distribution for my data?

If you need to estimate parameters, use MLE... but don't do that to decide between Pareto and lognormal.

Can you tell which of these is lognormal and which is Pareto?

enter image description here

Note that with the normal Q-Q plots (left column) we see the logs of data set 1 gives a fairly straight line, while data set 2 shows right skewness. With the exponential plots, the logs of data set 1 show a lighter right tail than exponential, while data set 2 shows a fairly straight line (the values in the right tail tend to wiggle about a bit even when the model is correct; this is not unusual with heavy-tails; it's one reason why you need to plot several samples of similar size to the one you're looking at to see what plots typically look like)

Code used to do those four plots:

qqnorm(log(y1))
qqnorm(log(y2))
qex <- function(x) qexp((rank(x)-.375)/(length(x)+.25))
plot(qex(y1),log(y1))
plot(qex(y2),log(y2))

If you have a regression type problem - one where the means change with other variables, you can really only assess the suitability of either distributional assumption in the presence of a suitable model for the mean.

| cite | improve this answer | |
$\endgroup$
3
$\begingroup$

This is a matter of model selection, of course, assuming that you just want to check whether your data comes from one model or the other and that your goal is not finding the right model among the infinite dimensional ocean of distributions. So, one option is to use AIC (which favours models with the lowest AIC value, and I will not attempt to describe here). Have a look at the following example with simulated data:

rm(list=ls())

set.seed(123)
x = rlnorm(100,0,1)

hist(x)

# Loglikelihood and AIC for lognormal model

ll1 = function(param){
if(param[2]>0) return(-sum(dlnorm(x,param[1],param[2],log=T)))
else return(Inf)
}

AIC1 = 2*optim(c(0,1),ll1)$value + 2*2

# Loglikelihood and AIC for Pareto model

dpareto=function(x, shape=1, location=1) shape * location^shape / x^(shape + 1)

ll2 = function(param){
if(param[1]>0 & min(x)> param[2]) return(-sum(log(dpareto(x,param[1],param[2]))))
else return(Inf)
}


AIC2 = 2*optim(c(1,0.01),ll2)$value + 2*2

# Comparison using AIC, which in this case favours the lognormal model.

 c(AIC1,AIC2)
| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Maybe fitdistr()?

enter image description here

require(MASS)
hist(x, freq=F)
fit<-fitdistr(x,"log-normal")$estimate
lines(dlnorm(0:max(x),fit[1],fit[2]), lwd=3)


> fit
meanlog     sdlog 
3.8181643 0.1871289 



> dput(x)
c(52.6866903145324, 39.7511298620398, 50.0577071855833, 33.8671245370402, 
51.6325665911116, 41.1745418750494, 48.4259060939127, 67.0893697776377, 
35.5355051232044, 44.6197404834786, 40.5620805256951, 39.4265590077884, 
36.0718655240496, 56.0205581625823, 52.8039852992611, 46.2069383488226, 
36.7324212941395, 44.7998046213554, 47.9727885542368, 36.3400338997286, 
32.7514839453244, 50.6878893947656, 53.3756089181472, 39.4769689441593, 
38.5432770167907, 62.350999487007, 44.5140171935881, 47.4026606915147, 
57.3723511479393, 64.4041641945078, 51.2286815562554, 60.4921839777139, 
71.6127652225805, 40.6395409719693, 48.681036613906, 52.3489622656967, 
46.6219563536878, 55.6136160469819, 62.3003761050482, 42.7865905767138, 
50.2413659137295, 45.6327941365187, 46.5621907725798, 48.9734785224035, 
40.4828649022511, 59.4982559591637, 42.9450436744074, 66.8393386407167, 
40.7248473206552, 45.9114242834839, 34.2671010054407, 45.7569869970351, 
50.4358523486278, 44.7445606782492, 44.4173298921541, 41.7506552050873, 
34.5657344132409, 47.7099864540652, 38.1680974794929, 42.2126680994737, 
35.690599714042, 37.6748157160789, 35.0840798650981, 41.4775827114607, 
36.6503753230464, 42.7539062488003, 39.2210050689652, 45.9364763482558, 
35.3687017955285, 62.8299659875044, 38.1532612008011, 39.9183076516292, 
59.0662388169057, 47.9032427690417, 42.4419580084314, 45.785859495192, 
59.5254284342724, 47.9161476636566, 32.6868959277799, 30.1039453246766, 
37.7606323857655, 35.754797368422, 35.5239777126187, 43.7874313667592, 
53.0328404605954, 37.4550326357314, 42.7226751172495, 44.898430515261, 
59.7229655935187, 41.0701258705001, 42.1672231656919, 60.9632847841197, 
60.3690132883734, 45.6469334940722, 39.8300067022836, 51.8185235060234, 
44.908828102875, 50.8200011497451, 53.7945569828737, 65.0432670527801, 
49.0306734716282, 35.9442821219144, 46.8133296904456, 43.7514416949611, 
43.7348972849838, 57.592040060118, 48.7913517211383, 38.5555058596449
)
| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Although this solution fits a lognormal to the data, it doesn't tell whether the fit is any good or whether a Pareto is a better choice. $\endgroup$ – whuber Feb 20 '19 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.