Building Confidence Intervals From Monte Carlo Simulations

Question

I have a discrete Markov process that I am running Monte Carlo simulations to estimate the distribution parameters, with the ultimate goal of producing a mean-time-to-failure (MTTF) estimate within a confidence interval.

I draw 10,000 samples using the Markov process to estimate failure times for each of 1,000 simulations. For each simulation, I fit a 2-parameter Weibull distribution to the failure time. I average the parameter values across all the simulations to get my final distribution estimates.

I understand the MTTF and variance gamma functions of Weibull distributions, but am unsure what n value to use for standard error and the best way to proceed.

1. Is the sample n value 10,000 (the number of samples in each simulation) or 1,000 (the number of simulations from which the distribution parameters were estimated)?

2. Or should I be using a normal distribution CI calculations because the parameter values across the 1,000 simulations are roughly normal?

3. Is it better to calculate a MTTF value for each simulation and perform a bootstrap estimate?

Edit: To clarify the problem setup:

I have a discrete Markov chain, with a state transition matrix of 1...m states, denoting the probabilities Pi,j that an item will degrade from the ith state to the jth state after a single interval. Lower-left values (i > j) of the matrix are zero, denoting the item cannot be upgraded. State 1 is less degraded than state 2, which is less degraded than step 3 etc.

To estimate the age of item failure, each item starts in state 1 (i = 1, a "new" condition). For a given state row (initially, state = 1, so row = 1), a randomly generated number from 0-1 is mapped to the transition column in the current state row that bounds the random value. This column, j, denotes its new state. Another randomly generated number is drawn and the process continues until the item reaches a "failed" state (where failure is predefined.)

Answer 1

This is a vague question without key specifics. Here is a correspondingly rough and vague outline that may be of some use.

Output from your Markov process may be very far from a sequence of independent observations from its limiting distribution. A thousand observations is a very small run, but perhaps useful. (If not, run your simulation program again to get something like 10,000 or 100,000 Markovian observations.)

• One can hope that after a hundred or so observations, a 'burn in' period, the process will have reached something like steady state. Discard an appropriate number of initial observations.

• Still, the remaining observations will typically not be independent. Make an autocorrelation plot with 30 or so lags, to find out how long it takes 'one-step' Markovian dependence to decay. Maybe you can view every 10th observation to be independent of others. Then 'thin' your sequence, after burn in, to use only every 10th observation. Maybe you will have something like $n=80$ remaining observations. One hopes they are nearly IID observations from the limiting distribution.

• Use these $n$ pseudo-IID observations to estimate parameters of whatever family of distribution you believe to include the limiting distribution of the Markov process.