1
$\begingroup$

For a simulation study I've been trying to find an appropriate distribution for job handling times in R. I have a very large dataset of 77010 records (handling time in seconds).

I've been exploring several distributions (lognormal, exponential, Gamma, Weibull, Burr, Pareto and more) and want to confirm which distribution best represents my data. I found that from all fitted distributions (using fitdist) the log normal distribution best represents my data (see picture below).

plot of fitdist with distribution lognormal

This plot makes me think that the lognormal distribution is a good representative of my data compared to the other distributions (also when comparing BIC, AIC & MLE). But, when I take the logs of my dataset and test for normality using ad.test (i.e., the Anderson Darling test), the null hypothesis is rejected. Likewise, when I perform the Kolmogorov-smirnov test (ks.test, note that ties are present here), the test is also rejected. I suspect my dataset is too large to obtain meaningful results from these tests.

How do I proceed in finding substantial evidence that this distribution fits my data?

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ The QQ plot does a good job in showing that the data distribution is extremely close to lognormal except in the upper tail. This has many important implications, because the moments of the lognormal are heavily influenced by the upper tail. For instance, using a lognormal distribution in your simulation is likely to produce simulated data whose moments are too large. What is the problem with just using the empirical distribution for your simulation? $\endgroup$
    – whuber ♦
    Sep 24, 2018 at 14:54

1 Answer 1

Reset to default
2
$\begingroup$

Often statistical tests for testing if your data following a specific distribution are not very informative. The best thing to do is what you did, namely, use a P-P or a Q-Q plot. These figures can give you a match better idea about the fit of the assumed distribution to your observed data, and potentially reveal problematic areas (e.g., the tails of the distribution).

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.