I would appreciate some advice on how to use Monte Carlo for estimating probabilities.

Generally speaking the problem I have involves running an experiment and counting the frequency of output (which depends on random inputs), and then using the relative frequency to estimate the probability of an output. However, the starting conditions is also randomly generated.

So I am repeatedly doing this:

  1. randomly generate starting conditions according to set rules,
  2. randomly generate input,
  3. record output.

My questions are:

  1. Is this a reasonable method?
  2. How can I estimate errors for the probability estimate?

Any help appreciated; I am new to this type of thing!

  • $\begingroup$ are your inputs and outputs discrete valued or continuous? How many different values can your inputs/outputs take? $\endgroup$
    – David Marx
    Sep 26 '13 at 23:25

I think it is a reasonable method. The difficulty is usually how to select the number of trials (which affects simulation time).

If you conduct $n$ trials, of which $N \leq n$ give a "positive" result, you can estimate the probability $p$ of positive result as its relative frequency,

$$\hat p = N/n.$$

This estimator is unbiased and has mean-square error, or variance,

$$\mathrm E[(\hat p-p)^2] = \mathrm E[(N/n-p)^2] = \frac{p(1-p)}{n},$$

as follows from noting that $N$ is a binomial random variable with parameters $n$, $p$. The root-mean-square (RMS) error, or standard deviation, is just the square root of this.

In order to assess whether an RMS error value is acceptable or not, that value should be compared with the true $p$. For example, an RMS of $0.01$ may be acceptable for estimating a $p=0.1$, but not for $p=0.001$ (the error would be ten times the true value). The usual approach for this is to normalize the error by dividing it by $p$. So the normalized RMS error is

$$\frac{\sqrt{\mathrm E[(\hat p-p)^2]}}{p} = \sqrt\frac{1-p}{np}.$$

This can be approximated as $1/\sqrt{np}$ for $p$ small. As you can see, to maintain a certain normalized error you need to simulate a number of times $n$ inversely proportional to $p$. But $p$ is unknown, so it's difficult to know which $n$ you need.

A solution is to use sequential estimation, in which $n$ is not fixed in advance, but is adaptively selected according to a certain stopping rule to guarantee that the estimation error does no exceed a predefined level. A standard sequential method is inverse binomial sampling (also called negative-binomial Monte Carlo), which consists in the following: continue simulating until a target number $N$ of positive results is achieved. So now $N$ is fixed, and the number of trials, $n$, becomes the random variable.

The nice aspect of this approach is that by selecting the target $N$ you can control the normalized error level that will be achieved, irrespective of the unknown $p$. That is, to each $N$ corresponds a certain guaranteed value of normalized error. This works whether you consider error defined as mean-square-error, mean-absolute-error, or in terms of a confidence interval. A description of the procedure in each case is given in the following papers (please bear with my self-referencing):

  • 1
    $\begingroup$ Could you explain why you divide the expectation by $p^2$ and how you then obtain the right hand side of the equation? $\endgroup$
    – whuber
    Sep 27 '13 at 13:19
  • $\begingroup$ Thanks for the suggestions. I have update my answer with these two points. The division is in order to normalize. Consider a root-mean-square error of 0.01. Would that be an accceptable value? Well, it depends on the true value of p: estimating a p=0.1 with 0.01 rms error is fine (10% error), but if p=0.001 then your error is ten times greater than the true value. So it's best to judge the error by normalizing it. The right-hand side of the expression is obtained in a strightforward way from the binomial distribution. $\endgroup$
    – Luis Mendo
    Sep 27 '13 at 13:35
  • $\begingroup$ Thank you for editing your answer to clarify this point: I'm sure many readers will appreciate your additional explanation. $\endgroup$
    – whuber
    Sep 27 '13 at 14:37
  • $\begingroup$ Welcome! Glad I could help $\endgroup$
    – Luis Mendo
    Sep 27 '13 at 16:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.