# How to calculate uncertainty of probability derived from random samples?

I'm running many simulations based on random samples, each of which produces a True or False result. The goal is to calculate the probability that the result will be True, which I can easily calculate using (number of True results / total number of tests).

As I increase the number of simulations, this probability value varies less and less, converging to a specific number that depends on the thing I'm testing. How do I calculate the "uncertainty" of this probability estimate as I run the simulations? In other words, as the number of simulations increases, this "uncertainty" number would get smaller and smaller, while the "probability" number varies less and less.

Do I have to do multiple trials of a certain number of simulations each, calculate the probability for each trial, and then measure the uncertainty of those probabilities as a group? Or is there a way to calculate it directly along with the overall average from running n simulations in a row?

In other words, the more simulations I run, the more accurate my result is. How do I calculate how accurate it is?

You can use a Confidence Interval to get a measure for the accuracy of your estimation.

Basically you are drawing from a Binomial distribution, i.e. your $$n$$ True/ False values correspond to $$X_i\sim Bi(p,1), \quad i=1...n$$. and you are trying to estimate $$p$$ by $$\hat{p}= \frac{1}{n}\sum_{i=1}^nx_i$$.

As $$n$$ increases you can use the Central limit theorem to approximate the distribution of $$\frac{1}{n}\sum_{i=1}^nX_i$$. So for a large enough $$n$$ it holds approximately that $$\frac{1}{n}\sum_{i=1}^nX_i \sim N(p, \sigma^2)$$. Now you can create a confidence interval with this approximation. This requires an estimation of the variance of the normal distribution by $$\hat{\sigma}^2= \frac{1}{n}\hat{p}(1-\hat{p})$$. The confidence interval then turns out to be $$[\hat{p}-z_\frac{\alpha}{2}\hat{\sigma}, \hat{p}+z_\frac{\alpha}{2}\hat{\sigma}]$$, where $$z_\frac{\alpha}{2}$$ is the $$\frac{\alpha}{2}$$-Quantile of the standard normal distribution. I.e if you want a $$0.95$$ confidence interval, $$z_\frac{\alpha}{2}$$ would be $$1.96$$.

The factor $$\frac{1}{n}$$ for the estimated variance (or $$\frac{1}{\sqrt{n}}$$ for the standard deviation) also explains why your accuracy get better for increasing $$n$$, as the confidence interval gets smaller and smaller.

What this method assumes is that

1. for each sample the probability of getting an overall true/false outcome is the same.
2. the samples are independent.
3. n is large enough for the approximation of the Central Limit theorem to work.