Explain confidence intervals in simple terms

Question

For one newly handling statistical analysis, please can you explain in the simplest of terms, with an example, what is a confidence interval?

Answer 1

There is a nice definition on Wikipedia, or here:

Confidence intervals are constructed at a confidence level, such as 95%, selected by the user. What does this mean? It means that if the same population is sampled on numerous occasions and interval estimates are made on each occasion, the resulting intervals would bracket the true population parameter in approximately 95% of the cases

But there is a discussion by Bayesians, e.g. Kruschke (2010, pp. 224-227), who state that classical CI's are commonly misinterpreted:

The 95% confidence interval consists of all values of $\theta$ that would not be rejected by a (two-tailed) significance test that allows 5% false alarms. (...) The confidence interval tells us something about the probability of extreme unobserved data values that we might have gotten if we repeated the experiment according to the covert intentions of the experimenter. But the confidence interval tells us little about the believability of any particular $\theta$ value, which is what we want to know.

Answer 2

I'll try with an example.

Call $X$ the length of the foot of a human being (in inches). It's not a constant so you can see it as a random variable. Assume this variable has a normal distribution $\mathcal{N}(9,1)$ : mean is 9 inches, standard deviation is 1 inch. (I don't mean to be very realistic :-)

The normal distribution is such that the probability that $X$ is in $[9-2;9+2]$ is (around) 95%. (2 is twice the standard deviation). If you meet somebody randomly in the street, the probability that his feet are between 7 and 11 inches long is 95%.

These are probabilities, not statistics : you know the distribution, you deduce probabilities about $X$.

Statistics is the other way around. Assume you know $X$ has a normal distribution $N(\mu,1)$ where the mean $\mu$ is unknown. You assume here the standard deviation is known (1 inch).

You meet somebody in the street, measure the length of his feet, and you want to know things about $\mu$ thanks to this observation.

You derive a confidence interval for $\mu$. You know that the probability that $X$ and $\mu$ are at a distance less than 2 is 95%. Write it mathematically :

$P(|X-\mu|\leq 2)=0.95$

Equivalently :

$P(\mu\in[X-2;X+2]])=0.95$

$[X-2;X+2]$ is the confidence interval for $\mu$ with confidence level 95%

Suppose you measured the guy's foot length to be 10 inches. Your confidence interval for $\mu$ is $[8;12]$. You say "the confidence interval for $\mu$ with level 95% is $[8;12]$".

You may wish to say simply "The probability that $\mu$ is between 8 and 12 is 95%". This would make things easy to explain. And honestly, it's the way we think intuitively. But it is not really true. Why it is not true is hard to explain and you may not care too much if you are not a mathematician. I won't get into it.

--

Now let's make it more sophisticated. You meet 10 people randomly in the street and measure the length of their feet: $X_1,X_2,...X_{10}$. Call $Y$ the average of these lengths : $Y=\frac{X_1+X_2+...X_{10}}{10}$. Mathematics tell you $Y$ has a normal distribution $\mathcal{N}(\mu,\frac{1}{\sqrt{10}}$). Using the same method, $Y$ and $\mu$ are at a distance less than $\frac{2}{\sqrt{10}}$ with probability 95%.

$P(\mu\in[X-\frac{2}{\sqrt{10}};X+\frac{2}{\sqrt{10}}])=0.95$

Thus your confidence interval for $\mu$ with level 95% is $[X-\frac{2}{\sqrt{10}};X+\frac{2}{\sqrt{10}}]$. Assume $Y=9.4$ empirically, then you have $[8.8;10.0]$ as a confidence interval. It is more precise since you have more information.

Again, you wish to say simply "$\mu$ is between 8.8 and 10 with probability 95%". It's ok as an intuition, false as a fact.

--

The way you created a confidence interval for $Y$ was very simple. Statistics provide further methods in more sophisticated situations.