Background: I'm comparing the performance of two algorithms. The data is noisy so the results tend to form a nice normal distribution.

I'd like to estimate the mean with a high confidence: up to 1% error with a 95% confidence.

As I collect samples, I start to get a better idea of the mean & stddev, but but how do I know when to stop?

  • $\begingroup$ The variance of the mean would be $s^2/\sqrt n$, where $s^2=\sum_i (x_i-\mu)^2/(n-1)$ $\endgroup$ – Aksakal Apr 18 '18 at 18:15

If you want the width of your confidence interval to have length no larger than 1 (0.5 on each side) then let's do some simple math.

The radius of a confidence interval, as a function of the sample size, is

$$ \dfrac{1.96 \cdot \sigma}{\sqrt{n}}$$

I'm using the z-statistic here instead of a t-statistic because in the limit, the two are equivalent (I'm also assuming you will need enough so that the difference between the two is negligible). We want this to be smaller than 1, so

$$ \dfrac{1.96 \cdot \sigma}{\sqrt{n}} \leq 0.5 \implies (2 \cdot 1.96)^2 \sigma^2 \approx 16 \times \sigma^2 \leq n $$

So a good rule of thumb is that you will need a sample size 16 times larger than the variance of your data in order to get a confidence interval with length 1 (0.5 on each side). This isn't a guarantee, but just a rule of thumb.

Here is a little plot of a simulation I performed. I construct 10 confidence intervals for varying levels of $\sigma$ and compute their lengths. I use the exact calculation ($n = (2 \cdot 1.96)^2 \sigma^2$ rather than the rule of thumb, which will result in smaller confidence intervals since $16>(2 \cdot 1.96)^2$). When the noise is small, our computation results in fewer samples, meaning the difference between using a z and t stat becomes important. As more noise is added, that difference becomes negligible.

enter image description here

If you want the width of the confidence interval to be some percent of the mean (e.g. the width of the confidence interval should be no larger than 1% of the estimate of the mean) then just replace $0.5$ with $0.005 \hat{\mu}$, where $\hat{\mu}$ is the estimate of the mean from the data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.