Parameter estimation of exponential distribution depending on multiple factors/attributes

for a simulation I’ll need to simulate project delays.

I’ve data on 20k projects with delays in quarters. Some of them were finished ahead of plan (i.e. neg. delays) others with a delay of 1-16 quarters. I assume a delay of 0 for everything finished ahead of plan as this does not get awarded in my case. The histogram looks like an exponential distribution would be a good fit.

I want to differentiate the distribution parameter (ie. lamda) depending on multiple attributes/factors of the project (3-5 factors, eg. region (south, north, west, east), project manager (X, Y, Z), type (basic, premium), etc.)). So I’m wondering what is the best approach to estimate the matrix of coefficients?

Individual estimation (for each combinations of factors):

• Should I extract subsamples for each possible combination of factors and estimate the lambdas in a regular way for each (eg MLE, LSE, etc.)?
• What must hold, that I could run the parametrization on each dimension (eg region) independently and then somehow cross the coefficient vectors to get the matrix? (rather then going through all combinations)
• Which test would you recommend to identify the 1-2 most sensitive dimensions/factors for the delay distribution (would ANOVA work assuming an exp. distribution and 5-8 groups each dimension)?

Multi-dimensional estimation:

• Any method allowing to run a multi-dimensional parametrization on the whole data set and will this be more accurate?

Many thanks for any input and ideas! Serge

If you're familiar with R (or Python, or many other statistical packages), using a sparse.model.matrix with a general model-builder like xgboost will automatically separate out the categories for variables and give you their relative importance.

If you actually need to encode the model somewhere else instead of just making predictions on a new data set, many generalized linear models can give you coefficients, and as long as you cast your data as a sparse matrix, those individual categories will survive.

• Many thanks! It's been a while since I was using R/Matlab myself but am taking a deeper look into it. Serge – Serge D Sep 15 '17 at 10:26

for a simulation I’ll need to simulate project delays.

I’ve data on 20k projects with delays in quarters. Some of them were finished ahead of plan (i.e. neg. delays) others with a delay of 1-16 quarters. I assume a delay of 0 for everything finished ahead of plan as this does not get awarded in my case.

The histogram looks like an exponential distribution would be a good fit.

Presumably that's the marginal distribution you're looking at; this can be misleading, since what you're trying to discuss is the conditional distribution (conditional on the attributes you mention)

Also being quarterly, it's discrete (or, arguably interval-censored) rather than continuous. The exact zeroes may be an issue because they probably don't fit with the patter of the rest; you might perhaps need to consider zero-inflated distributions

I want to differentiate the distribution parameter (ie. lamda) depending on multiple attributes/factors of the project (3-5 factors, eg. region (south, north, west, east), project manager (X, Y, Z), type (basic, premium), etc.)). So I’m wondering what is the best approach to estimate the matrix of coefficients?

Likely a generalized linear model (or similar) with a conditional distribution suited to the fact that times will be right skew and likely heteroskedastic. Perhaps a negative binomial or a zero-inflated gamma model.

Should I extract subsamples for each possible combination of factors and estimate the lambdas in a regular way for each (eg MLE, LSE, etc.)?

No, generally it's better to do it all in one model

What must hold, that I could run the parametrization on each dimension (eg region) independently and then somehow cross the coefficient vectors to get the matrix? (rather then going through all combinations)

This won't work in general. e.g. see Simpson's paradox

Multi-dimensional estimation:

Any method allowing to run a multi-dimensional parametrization on the whole data set and will this be more accurate?

See the above suggestion.

• Thank you! The zero-inflated distribution is a good path. An approach I am currently looking at: * adjust for zeros * for 3 factors calculate the avg. delay (only on delayed projects) for the respective combination * smooth the means with a lin. regression (to avoid big jumps as some sub-samples are rel. small) * take the smoothed means as MLE for the Exp- distribution – Serge D Sep 15 '17 at 11:08