# Does a confidence interval carry some extra error for non perfectly normal distributions?

I'm not trying to nitpick but I always want to make sure with stats that I clearly understand the delimitation between exact/theoretical measurements and "real life" measurements.

Let's say for example that our population is students at a university and we're interested in their heights. We randomly sample $100$ out of $10,000$ students and take the mean of that sample.

When we talk about a $95 \%$ confidence interval for that sample mean, I understand it as saying "given that we assume a normal distribution for the sampling distribution of the sample means of samples of size $n = 100$, there's a $95 \%$ chance that the mean of our sample falls at a point under our sampling distribution (normal) curve such that the true mean of the population is going to be within $2$ standard deviations of our sample mean". (is this correct?)

However, it seems to me that because of the assumption of perfect normality in the use of area measurements for a certain number of SD's, our $95 \%$ confidence interval carries the extra error between a true normal distribution and our actual distribution, so that in that sense it's not a true $95 \%$ confidence interval but a $95 \%$ confidence interval only once we assume perfect normality, which means that in reality we might be off more often than $5 \%$ of the time.

Is this correct or am I missing something?

• I would like to re-emphasize something you stated correctly: the Normality assumption pertains to the sampling distribution of the means, not to the heights in the population. The sampling distribution of the means in this example is going to be very close to Normal. What a good statistician does is akin to what astrophysicists do when they use Newtonian calculations to compute distances to bodies in the solar system: although they know that model isn't correct, they also know how to determine when it's worth going to the extra work of using the more accurate equations of General Relativity. – whuber Jul 21 '16 at 14:19
• @whuber, thank you for your comment. I can really start to see that there's truly an art to deciding when to use what in statistics. I assume it takes both a lot of experience and a strong theoretical grasp of the math behind the concepts. – jeremy radcliff Jul 21 '16 at 18:55

## 2 Answers

When we talk about a 95% confidence interval for that sample mean, i understand it as saying "given that we assume a normal distribution for the sampling distribution of the sample means of samples of size n=100, there's a 95% chance that the mean of our sample falls at a point under our sampling distribution (normal) curve such that the true mean of the population is going to be within 2 standard deviations of our sample mean". (is this correct?)

This is not quite correct, and is one of the most common misconceptions in statistics. You can find more discussion here. The 95% confidence interval means that we are 95% confident that the parameter lies in our interval. Another way to think about this is, if we were to repeat the sampling many more times, and create confidence intervals every time, then approximately 95% of those confidence intervals will contain the true parameter.

Otherwise, you are correct that when we appeal to the CLT in this way (in assuming normality of the sample mean), there is an added error since the distribution is never going to be exact. However, our confidence in the estimate might either go up or down.

Here is a quote from Tsou and Royall(1995)

A popular $95\%$ confidence interval for $E(X)$ based on $n$ observations is the $t$ interval, $\bar{x} \pm t_{n-1} sn^{-1/2}$, where $s^2 = \sum (x - \bar{x}^2)/(n-1)$. This is actually a $95\%$ confidence interval if the $X$'s are iid $N(\theta, \sigma^2)$. But if this model is incorrect, then it is no longer true that the coverage probability equals the nominal confidence coefficient, .95.

I am going to simulate this behavior in the following R code. I draw samples of size $N = 50$ from first a Normal(5, 1) population and then a $t_{1}$ distribution with mean $\mu = 5$. In the first case, since the population is truly normal, the distribution of the sample mean is exactly normal. In the second case the distribution is a shifted $t_1$ distribution which has much longer tails than the normal distribution. For each of these I simulate a sample of size $N = 50$, 1000 times, make the confidence interval for each time, and check whether $\mu = 5$ is in the interval or not. I return the proportion of time $\mu$ was in the interval, and this number is expected to be .95 if all assumptions hold.

set.seed(100)
## True value of mu
mu <- 5
reps <- 1000 # to demonstrate definition of CI
N <- 50

counting <- vector(length = reps)
## making 95% CI
for(i in 1:reps)
{
## When data is really normal and true mean is mu = 5
## So CLT hold exactly
data <-  mu + rnorm(N, mean = 0, sd = 1)
mu.hat <- mean(data)
se <- sd(data)/sqrt(N)
quantile <- qt(.975, df = N-1)
upper <-  mu.hat + quantile*se
lower <- mu.hat - quantile*se

## Demonstrating how many of the CIs  have mu in them

counting[i] <- ifelse(upper > mu && lower < mu, 1, 0)
}
mean(counting)
#  0.946

counting <- vector(length = reps)
## making 95% CI
for(i in 1:reps)
{
## When data is from t distribution and true mean is still mu = 5
## With N = 100 CLT is only approximate
data <-  mu + rt(N, df = 1)
mu.hat <- mean(data)
se <- sd(data)/sqrt(N)
quantile <- qt(.975, df = N-1)
upper <-  mu.hat + quantile*se
lower <- mu.hat - quantile*se

## Demonstrating how many of the CIs  have mu in them

counting[i] <- ifelse(upper > mu && lower < mu, 1, 0)
}
mean(counting)
#  0.986


The first time, I get very close to .95 but the second time I am much higher. So yes, our confidence will be different from .95 if our assumptions don't hold. However if $N$ is large and the data distribution is close to normal, it won't be too far off.

• Thank you very much for taking the time to go into such detail (is the code in R?). Would it be correct then to say that "There's a 95% chance that the sample we randomly chose is one of the samples whose mean is within 2 SD's of the true mean"? In other words, 95% of all the possible samples of size $n$ are going to capture the true mean. – jeremy radcliff Jul 18 '16 at 14:34
• Yes, the code is in R. Still a little off, since "samples" don't capture mean, confidence intervals do. So maybe, "95% of all the possible CIs created from samples of size n are going to capture the true mean", would be more accurate. – Greenparker Jul 18 '16 at 14:36
• @Greenparker, that makes sense thank you, I think I understand the subtle distinction now: We're saying that once we have our sample we can't talk about the probability of the CI created from it capturing the mean because at that point the CI either captures the true mean or doesn't but there's no chance associated with that possibility anymore since it is already "realized"; however we can talk about the probability of getting one of the samples whose CI will capture the true mean. – jeremy radcliff Jul 18 '16 at 14:40

The answer by @Greenparker is an excellent answer that highlights a common misconception, but I want to try to address your question more directly. You are correct to say that if your model is wrong (student heights are not normal) then your confidence intervals might not be as accurate as they claim. The issue then lies with identifying an appropriate model and then constructing confidence intervals from that model for the parameters involved (confidence intervals are not something tied solely to models with the normality assumption).

In some cases, even if your model is wrong, your statistics may be robust to deviations from your model assumptions (e.g. The normal assumption for your heights was wrong, but you end up with 94% confidence intervals rather than 95% confidence intervals...which is just fine).

And of course as @Greenparker illustrates, the Central Limit theorem is very powerful and holds in many situations. So if your sample size is large enough you can often get away with Normality assumptions.

All this to say that statistics is quite nuanced, problem specific.

• Thank you for adding this explanation. I guess in that sense stats is like any other math modeling problem; you have a rigorous formally defined base and then there's an art to making that work "in the real world". Hopefully I'll keep building a better sense for that transition. – jeremy radcliff Jul 19 '16 at 13:52