0
$\begingroup$

I'm trying to build a simulation for a quality control process, where quality analysts inspect the product and report faults if they found any.

I have a dataset of this bug reports, so I'm trying to fit the inter-arrival time of this report to a probability distribution. All the literature I found mentioned that an exponential distribution would be a would choice, however I got this results while applying the Anderson-Darling test:

# This is using Python's Scypy library
statistic, critical_values, significance_level = stats.anderson(data_series, "expon")

Output:

Anderson-Darling Test for  expon : statistic  370.733327629
Critical Value:  0.921  Significance Level:  15.0
Critical Value:  1.077  Significance Level:  10.0
Critical Value:  1.339  Significance Level:  5.0
Critical Value:  1.604  Significance Level:  2.5
Critical Value:  1.954  Significance Level:  1.0

Which seems to agree with what I obtain with Kolmogorov-Smirnov:

#cdf_function is the exponential distribution fitted using Maximum Likelihood
d, p_value = stats.kstest(data_series, cdf_function)

Output:

Kolmogorov-Smirnov Test for  expon : d  0.38586872273  p_value:  0.0

So, apparently my dataset is not a good fit for the exponential distribution. Which other distributions are used for inter-arrival time modelling?

UPDATE: Here is an histogram of the inter-arrival times found on the dataset. The y-axis is in hours.

Inter-arrival time for fault reports, in hours

SECOND UPDATE: The histogram above corresponds to the inter-arrival time (in hours) for fault reports by the most productive quality analyst of the team, during a 20 month period. If we apply the base-10 log to the inter-arrival time, the histogram is the following:

Base-10 log of inter-arrival time

$\endgroup$
  • 1
    $\begingroup$ I would not expect the intervals between bug reports to be exponential -- they'll tend to come in clusters after new features are added, and then reduce as bug fixes are released. They might also be affected by day of week effects and time of year effects and so on. It's just possible they might be reasonably approximated by a mixture of exponentials perhaps, but why do you need a specific distributional model for the times? $\endgroup$ – Glen_b Jul 10 '16 at 16:22
  • $\begingroup$ @Glen_b I'm trying to build a discrete-event simulation model for the bug reporting process. From what I read, I need to use a "theoretical" probability distribution in my model that fits the data extracted from the real process operation $\endgroup$ – Guillermo Guardastagno Jul 10 '16 at 20:47
  • 1
    $\begingroup$ One possibility is to smooth that observed distribution and sample that (e.g. kernel smoothed densities are quite easy to sample from). Alternatively, it may be that a transformed version of the data is easier to smooth (and then back-transform). A third possibility is some simpler mixture distribution where the mixture components are right-skew (possibly a mix of exponentials). Are there exact zeros in your data? If not, what do the logs of the times look like? $\endgroup$ – Glen_b Jul 10 '16 at 23:42
  • 1
    $\begingroup$ Another possibility is to slice the original events into bins and model the counts to get event-rates, which would then lead to (again) a mix of exponentials. $\endgroup$ – Glen_b Jul 10 '16 at 23:49
  • 1
    $\begingroup$ Actually a truncated (on the left) gamma with a very small shape parameter looks like a fairly good approximation, I'll try to come back and post some details $\endgroup$ – Glen_b Jul 11 '16 at 15:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.