2
$\begingroup$

I'm running many simulations based on random samples, each of which produces a True or False result. The goal is to calculate the probability that the result will be True, which I can easily calculate using (number of True results / total number of tests).

As I increase the number of simulations, this probability value varies less and less, converging to a specific number that depends on the thing I'm testing. How do I calculate the "uncertainty" of this probability estimate as I run the simulations? In other words, as the number of simulations increases, this "uncertainty" number would get smaller and smaller, while the "probability" number varies less and less.

Do I have to do multiple trials of a certain number of simulations each, calculate the probability for each trial, and then measure the uncertainty of those probabilities as a group? Or is there a way to calculate it directly along with the overall average from running n simulations in a row?

In other words, the more simulations I run, the more accurate my result is. How do I calculate how accurate it is?

$\endgroup$

1 Answer 1

2
$\begingroup$

You can use a Confidence Interval to get a measure for the accuracy of your estimation.

Basically you are drawing from a Binomial distribution, i.e. your $n$ True/ False values correspond to $X_i\sim Bi(p,1), \quad i=1...n$. and you are trying to estimate $p$ by $\hat{p}= \frac{1}{n}\sum_{i=1}^nx_i$.

As $n$ increases you can use the Central limit theorem to approximate the distribution of $\frac{1}{n}\sum_{i=1}^nX_i$. So for a large enough $n$ it holds approximately that $\frac{1}{n}\sum_{i=1}^nX_i \sim N(p, \sigma^2)$. Now you can create a confidence interval with this approximation. This requires an estimation of the variance of the normal distribution by $\hat{\sigma}^2= \frac{1}{n}\hat{p}(1-\hat{p})$. The confidence interval then turns out to be $[\hat{p}-z_\frac{\alpha}{2}\hat{\sigma}, \hat{p}+z_\frac{\alpha}{2}\hat{\sigma}]$, where $z_\frac{\alpha}{2}$ is the $\frac{\alpha}{2}$-Quantile of the standard normal distribution. I.e if you want a $0.95$ confidence interval, $z_\frac{\alpha}{2}$ would be $1.96$.

The factor $\frac{1}{n}$ for the estimated variance (or $\frac{1}{\sqrt{n}}$ for the standard deviation) also explains why your accuracy get better for increasing $n$, as the confidence interval gets smaller and smaller.

What this method assumes is that

  1. for each sample the probability of getting an overall true/false outcome is the same.
  2. the samples are independent.
  3. n is large enough for the approximation of the Central Limit theorem to work.
$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.