Can someone give me details about a particular confidence interval formula?

Question

I'm trying to fully understand the confidence interval formal given on this site:

$$\hat{\mu}\pm z_{1-\alpha/2}\sqrt{\frac{\hat{\mu}(1-\hat{\mu})}{n}}$$

so I can reproduce the same type of intervals for my own data. But I don't quite understand what the parameters such as $\alpha$ and $Z$ mean. I'm guessing they're related to defining a 95% confidence interval if your data were distributed normally.

Can explain to me how that formula works or a reference I can find a description of the formula used?

Answer 1

If $\hat{\mu}$ is the mean error rate computed averaging $N$ error rates from different $N$ tests, an explanation could be:

Let $X$ be the number of errors on $N$ tests, so $X$ is a binomial distributed random variable with mean $N\hat{\mu}$ and variance $N\hat{\mu}(1-\hat{\mu})$ (it is sum of $N$ Bernoulli random variables).

Thus $X/N\sim Bin\bigg(\hat{\mu},\frac{\hat{\mu}(1-\hat{\mu})}{N}\bigg)$.

By the central limit theorem it could be approximated to a normal random variable with same mean and variance. Then you can compute the $\alpha$ confidence interval with:

$$P\bigg(-z_{1-\alpha/2}\leq\frac{\mu-\hat{\mu}}{\sqrt{\hat{\mu}(1-\hat{\mu})/N}}\leq z_{1-\alpha/2}\bigg) = 1 - \alpha$$

Bibliography: It is similar to estimate a confidence interval for accuracy using a $N$ values test set in a classification problem. You should take a look to P.N. Tan, M. Steinbach, V. Kumar Introduction to Data Mining. Addison Wesley, 2006.

Answer 2

Answering the following part of your question:

I don't quite understand what the parameters such as alpha and Z mean

$\alpha$ is the parameter that defines the confidence level of the interval. Specifically, the confidence level will be $100(1-\alpha)$%, so to get a 95% confidence interval, set $\alpha=0.05$.

$Z$ is a reference to the normal distribution, and in this case $z_q$ means its $q$-th quantile, that is the value for which $P(Z < z_q) = q$, where $Z$ is the standard normal distribution. This can be looked up in tables or calculated by computers. For example, when $\alpha=0.05$, the formula needs the 0.975-th quantile, that is the value which exceeds 97.5% of the normal distribution. Its value is $z_{0.975}=1.96$.