Can someone explain what is the confidence interval? Why it should be 95%? When it is used and what is it measuring? I understand it's some kind of evaluation metric, but I can't seem to find a decent explanation by connecting it to real-world examples.
Any help would be greatly appreciated.
thanks
A confidence interval can be used whenever you estimate a population parameter and there is uncertainty involved and takes the form $\text{estimate} \pm \left(\text{critical value} \times \text{standard error}\right)$. So instead of a point estimate you report a range over which you believe the parameter's values can lie (centered at the point estimate). For example, if you want to estimate the weights of people in a city, you could sample $1000$ people and take the average weight to get a number. But if you wanted a range you would create a confidence interval. It doesn't have to be $95\%$. Other popular choices are $99\%, 90\%$. But if we found that the mean weight in the sample was $120$ pounds and a $95\%$ confidence interval was found to be $(115, 125)$, we'd say that we are $95\%$ confident that the true population mean lies in this interval. That is if we created a lot of $95\%$ confidence intervals, about that percentage of them would contain the true population mean. A $90\%$ confidence interval will always be narrower than a $95\%$ confidence interval.
A confidence interval is a set estimator used in statistics to give you a broad estimated region for an unknown parameter, with a specified confidence level. A confidence interval may be formed at any confidence level (not just 95% confidence) and it will give you a set of values that you think includes the parameter of interest. When you form a confidence interval from data, you can then say things like "we find that such-and-such parameter falls within this interval with 95% confidence".
The basic mathematical theory of a confidence interval is as follows. Suppose you have a statistical problem with an unknown parameter $\theta \in \Theta$ and you will observe data $\mathbf{x}$ relating to that parameter (i.e., this data has a sampling distribution that depends on the unknown parameter). To form a confidence interval you choose a confidence level $1-\alpha$ and you form an interval that depends on the (random) data such that the parameter $\theta$ will fall within the (random) interval with probability equal to the specified confidence level.
To formalise this idea, let $\mathscr{X}$ denote the set of possible outcomes for the data and let $\mathscr{T}$ denote the class of all possible subsets of the parameter space $\Theta$. Mathematically, a confidence interval is a set function $\text{CI}: \mathscr{X} \times [0,1] \rightarrow \mathscr{T}$ that meets the following probability requirement:
$$\mathbb{P}(\theta \in \text{CI}(\mathbf{X}, \alpha) |\theta) = 1-\alpha \quad \quad \quad \text{for all } \theta \in \Theta.$$
This probability requirement means that, regardless of the true value of $\theta$, this parameter value will fall within the (random) interval $\text{CI}(\mathbf{X}, \alpha)$ with the stipulated probability $1-\alpha$. Note that once we observe the data $\mathbf{x}$ we substitute this to obtain the (fixed) confidence interval $\text{CI}(\mathbf{x}, \alpha)$, which is no longer random.
In practice, there are many different kinds of confidence intervals used in statistics for different types of statistical problems. For example, we may obtain a confidence interval for an unknown population mean, population variance, unknown correlation between random variables, etc. Different types of confidence intervals have different derivations and formulae, but they should all obey the above probabilty requirement. (In some cases they obey this requirement only under a simplifying approximation such as use of an asymptotic distribution.) Often, confidence intervals are formed using pivotal quantities, which can be used to give probability statements of the above form.
If you would like to learn more about confidence intervals, I recommend you have a look at the specific forms and derivations for confidence intervals for particular types of parameters, such as estimating a population mean, variance, etc. There are a number of practical questions on forming confidence intervals on the statistics site CV.SE (see the confidence-interval tag here).
