4
$\begingroup$

The Unknown Distribution

I have a dataset whose histogram showed above (the blue part), and I want to scale it for later machine learning process so I am trying to do a parameter estimation.

Its histogram shows that looks like a gamma distribution, then I tried gamfit in matlab to get the fit curve(showed in orange). It's obviously that this curve does not fit the data very well so I decided to look for other distribution.

I found a page on wikipedia called List_of_probability_distributions and there are just too many of them. One of my friends says that there is a general way to determine the distribution of certain data but she doesn't know how to do it. Is there any guidance to do that?

$\endgroup$
  • $\begingroup$ I don't have time to check, but I guess many good answers to this very well-know problem can easily be found on this website or on the web. It comes down in the end to fitting some distribution to some data. You can learn the parameters with the method of moments, maximum likelihood, etc. Statistical tests, heuristics, nonparametric approaches can be used for goodness of fit... This is tackled in any undergraduate stats textbook $\endgroup$ – Antoine Jun 8 '16 at 10:09
  • $\begingroup$ I wrote about this problem elsewhere. There, I use R to illustrate the process. $\endgroup$ – COOLSerdash Jun 8 '16 at 10:45
3
$\begingroup$

The gamma indeed already looks like a good bet, and your plot indeed shows that, in fact, it isn't.

The way the red line lies to the right of the falling flank of the histogram indicates that you have a heavy tail, that is, more large values than would be expected from a gamma. So we should look for a distribution that is similar to the gamma (positive support, asymmetric etc.) but allows for a larger variance.

The "related distributions" section of the Wikipedia article on the gamma distribution is a good place to start. It indicates the generalized gamma distribution as a, well, generalization of the "normal" gamma. It has three parameters, compared to the gamma's two, so it should be more flexible and be better able to fit your data. You appear to have enough data to reasonably estimate three parameters, too.

$\endgroup$
0
$\begingroup$

Thanks Stephan I worked it out. I choose R's package flexsurv & fitdistrplus to fit the curve. Here is how I worked it out:

library('flexsurv')
library('fitdistrplus')

fitres <- fitdist(datavector, "gengamma",method = 'mle', start=list(mu=log(5),sigma=log(1.2),Q=1)) 

#If you want to plot the curve, then

est = fitres$estimate
H = hist(res,breaks = 500)
plot(H, freq = FALSE, ylim=c(0,max(H$density*1.2)))
curve(dgengamma(x,est[1],est[2],est[3]), col=2, lwd=2,xlim=range(res), add=TRUE)

Pretty simple.

If you have something else to do with python, save your code to a file, then use rpy2 to call R process, call source to load the file. Use multiprocessing.Pool to perform a multiprocessing task, ThreadPool won't do.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.