How to determine the distribution of dataset?

Question

I have a dataset whose histogram showed above (the blue part), and I want to scale it for later machine learning process so I am trying to do a parameter estimation.

Its histogram shows that looks like a gamma distribution, then I tried gamfit in matlab to get the fit curve(showed in orange). It's obviously that this curve does not fit the data very well so I decided to look for other distribution.

I found a page on wikipedia called List_of_probability_distributions and there are just too many of them. One of my friends says that there is a general way to determine the distribution of certain data but she doesn't know how to do it. Is there any guidance to do that?

Answer 1

The gamma indeed already looks like a good bet, and your plot indeed shows that, in fact, it isn't.

The way the red line lies to the right of the falling flank of the histogram indicates that you have a heavy tail, that is, more large values than would be expected from a gamma. So we should look for a distribution that is similar to the gamma (positive support, asymmetric etc.) but allows for a larger variance.

The "related distributions" section of the Wikipedia article on the gamma distribution is a good place to start. It indicates the generalized gamma distribution as a, well, generalization of the "normal" gamma. It has three parameters, compared to the gamma's two, so it should be more flexible and be better able to fit your data. You appear to have enough data to reasonably estimate three parameters, too.

Answer 2

Thanks Stephan I worked it out. I choose R's package flexsurv & fitdistrplus to fit the curve. Here is how I worked it out:

library('flexsurv')
library('fitdistrplus')

fitres <- fitdist(datavector, "gengamma",method = 'mle', start=list(mu=log(5),sigma=log(1.2),Q=1)) 

#If you want to plot the curve, then

est = fitres$estimate
H = hist(res,breaks = 500)
plot(H, freq = FALSE, ylim=c(0,max(H$density*1.2)))
curve(dgengamma(x,est[1],est[2],est[3]), col=2, lwd=2,xlim=range(res), add=TRUE)

Pretty simple.

If you have something else to do with python, save your code to a file, then use rpy2 to call R process, call source to load the file. Use multiprocessing.Pool to perform a multiprocessing task, ThreadPool won't do.