Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm trying to fit a data sample to a distribution. So far I have created a histogram and fitted the data with a lognormal distribution in R and made a Q-Q plot in excel (of log(benefits paid) against theoretical normal quantiles).

Here are my histogram and Q-Q plot: enter image description here enter image description here

However, I don't think this distribution is fitting the data closely enough.

I was hoping for an opinion on whether there are any distributions which should fit the data better. Or is this as closer fit as I'm going to be able to get?

share|improve this question
    
This may be a moot point, but why did you use excel to create the q-q plot, instead of using R's qqplot()? –  Mark Ramotowski Apr 18 at 12:55
    
I was trying to avoid using R because I didn't know how but now I do so I can do that now! –  user135784 Apr 21 at 11:17
    
It is so much easier! Happy days! –  Mark Ramotowski Apr 21 at 11:19
    
lol true thanks!! –  user135784 Apr 21 at 11:20
    
What exactly is the relationship between this histogram and the Q-Q plot? The distribution depicted in the latter (which is approximately symmetric and very short-tailed) very clearly is not at all like the former (which is highly asymmetric with a long right tail). Although you indicate the Q-Q plot concerns log benefits paid, the values (which range approximately between -2 and 2) are not logarithms of the data in the histogram. –  whuber Jun 5 at 13:19
add comment

2 Answers 2

up vote 2 down vote accepted

The bins in your histogram are a bit too wide (the default has too few bins), making it hard to discern the shape clearly.

The QQ-plot suggests that a finite mixture of perhaps two or three components, possibly of something right skew, perhaps like gamma distributions.

share|improve this answer
    
Thanks Glen! Yes I see what you mean I tried to set the number of bins lower however unfortunately R automatically altered this and I'm not skilled enough to change this. I had a look on Wikipedia and found a distribution called the double-pareto lognormal which looks as though it would be appropriate but I'm not really skilled enough to use that either. Thanks for the further insight though I will keep this in mind! –  user135784 Apr 18 at 10:14
    
(1) I wanted more, not fewer bins! (2) If you're using the hist command I suspect you made an error. I use it often, and the number of bins is trivial to change: try x=rgamma(999,2.5,.25) to make some data, and then hist(x, n=40) which will give close to 40 bins, as an example. If you're using some other command to make histograms you'll need to look at the help. –  Glen_b Apr 19 at 0:04
    
Your data has a hint of several right skew modes, which is why I suggested a mixture of unimodal distributions. Few right skew distributions that you can easily find code to generate are shaped like your sample. –  Glen_b Apr 19 at 0:07
    
Thanks Glen! I'll try to adjust the number of bins up then. Sorry about that! Ah ok I'll try fitting the data like you said with multiple unimodal distributions although I'm pretty new to R so if you have any tips on how I can do this that would help. Thanks!! –  user135784 Apr 19 at 2:44
    
Depending on your needs, in simple cases it can be done reasonably well by hand. There are some packages that are useful for mixture models, some use EM some use other approaches. A number of packages for mixtures of Gaussians exist, so you could take logs and try one of those first. Indeed if I was fitting gammas I'd do the same to try to spot the bumps in the log-density to figure out where to initialize my fits. –  Glen_b Apr 19 at 5:29
show 3 more comments

Have you tried a gamma distribution? That is a little more flexible than a log-normal and typically has thinner tails, which appear to be where your fit is lacking.

If that doesn't work, you might want to click through this list on wikipedia. There are many choices for distributions with positive support, and of course each one has it's pluses and minuses depending on your application. It looks like you're doing something with financial data, so the Pareto distribution may also be worth exploring. That tends to have thick tails though so maybe try the gamma first.

A piece of advice though: you're not going to find a closed-form distribution that fits your data perfectly. Different distributions will miss in different ways, so you'll need to ask yourself exactly how good and where in the distribution the fit has to be. The log-normal fit in your pictures may not actually be that bad depending on your priorities. If you're doing regression analysis, then maybe it works. If you're modeling tail-risk, then maybe not. Also, do you really need a closed form distribution? If distribution fit qua fit is your goal, then maybe the empirical distribution is the way to go.

share|improve this answer
    
Out of curiosity, what is the basis for saying that gamma distributions are more flexible than lognormal? –  Glen_b Apr 18 at 7:46
1  
Well really I mean that its shape can take more obviously different forms depending on that shape parameter than the log-normal, which is at least built from a self-similar location-scale family (the normal). Technically they both have two parameters so it's not that much more flexible, but in a narrow sense... There is also a distribution with an additional parameter, which I forget right now, that includes the gamma as a special case. –  Mike Nute Apr 18 at 8:03
    
Thanks Mike! Your answer is very helpful. I'll try out the Gamma and Pareto distributions and skim through wikipedia and decide which one is best based on your advice! –  user135784 Apr 18 at 8:35
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.