2
$\begingroup$

I'm doing some research (I haven't studied anything related to statistics) and I have been stuck for a while now so bear with me:

I am trying to simulate data (generate data) based on empirical data. My empirical data is a list of 4276 numbers representing travel times ranging from 143 to 596 minutes. The available data is discrete in the sense that minutes is the smallest time precision. (everything is rounded to minutes).

You can see the estimated PDF (normalized frequency) distribution below.

My data

The problem is: I want to generate data in a discrete simulation according to the reality above. So I would like to fit a distribution to this to be able to reproduce data according to that distribution.

Now, I tried inputting the data in Arena's input analyzer and the best fit is a Gamma distribution.

I also tried fitting a Poisson curve as stated in this thread without any result:

Poisson fit

Here you can see the Gamma fit Arena's input analyzer found:

Arena's best fit

Now I can imagine if I reproduce data starting from that Gamma distribution it will not meet the reality.

How would I go about fitting this data set where traveltimes under 350 (approx.) are represented much more in frequency than traveltimes above 350.

Am I allowed to group data > 350 to get a normal distribution or something like that?

I'm not really sure which way to go from here because I have not studied anything in that field and I am quite lost.

Another path I was exploring was: Keep simulating data until the simulated data would approximate the empirical data. I mean I would keep simulating until my $\chi^2$ of the generated and empirical data would be acceptable. But that was a bad idea because I am having multiple distributions in my simulation which resulting in excessive long simulation time...so not realistic.

I am trying to do all my analysis in Python. However, I did try the input analyzer from Arena to confirm results.

The raw data is available here: Pastebin

Hope this is even possible and I am eager to hear from you guys!

$\endgroup$
1
$\begingroup$

Ok, so you want to fit a reasonable$^{1}$ density to your data and then generate random values from its distribution. You consider Poisson or Gamma distributions as reasonable for your data.

A Poisson distribution is fully determined by its mean, $\lambda$, or equivalently its rate, $\frac{1}{\lambda}$. (I believe you switched these parameterizations in your first plot.)

In any case, I find $\lambda = 236.8167$ for your data.

The Gamma distribution has a couple of different parameterizations, one of which is with a shape ($\alpha$) and rate ($\beta$) parameter. For your data, I find: $\alpha = 50.5633$ and $\beta = 0.2135$.

The following plot lays the fitted Poisson and Gamma densities over a histogram of the raw data.

Histogram with densities

Eye-balling the plot, I would conclude that a Gamma distribution is a good fit and would be a better choice than Poisson.

In Python, numpy.random.gamma provides a sampler. Note that it parameterizes the Gamma distribution with shape ($k$) and scale ($\theta$) parameters. The scale parameter $\theta $ is equal to $\frac{1}{\beta}$.


${1.}$ Reasonable in the sense of coming from prior or domain knowledge. A left-truncated normal may not be a bad choice either in terms of fit, but could possibly be ruled out on other grounds.

$\endgroup$
  • $\begingroup$ Thank you very much for your answer! I managed to fit a gamma aswell and generate data from it. I tried running some fitting tests on it but without success. How can I prove that this distribution is actually pretty good? Or is visual fitting enough and no need to prove it with numbers? $\endgroup$ – Doesntmatter Mar 20 '18 at 17:41
  • 1
    $\begingroup$ @Doesntmatter couple of things. 1) Kolmogorov-Smirnov tests provide p-value < 2.2e-16 for Poisson and p-value = 0.006529 for the Gamma. 2) As a matter of fact in statistics: you can't prove that these numbers are generated by a given distribution. You can reject the null hypothesis that they were generated by a Poisson distribution for example. 3) More formal eye-balling could/should be done using QQplots. 4) Strongest would be if you can argue from domain knowledge that it must be a certain distribution. $\endgroup$ – Jim Mar 21 '18 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.