Does a confidence interval carry some extra error for non perfectly normal distributions? I'm not trying to nitpick but I always want to make sure with stats that I clearly understand the delimitation between exact/theoretical measurements and "real life" measurements.
Let's say for example that our population is students at a university and we're interested in their heights. We randomly sample $100$ out of $10,000$ students and take the mean of that sample.
When we talk about a $95 \%$ confidence interval for that sample mean, I understand it as saying "given that we assume a normal distribution for the sampling distribution of the sample means of samples of size $n = 100$, there's a $95 \%$ chance that the mean of our sample falls at a point under our sampling distribution (normal) curve such that the true mean of the population is going to be within $2$ standard deviations of our sample mean". (is this correct?)
However, it seems to me that because of the assumption of perfect normality in the use of area measurements for a certain number of SD's, our $95 \%$ confidence interval carries the extra error between a true normal distribution and our actual distribution, so that in that sense it's not a true $95 \%$ confidence interval but a $95 \%$ confidence interval only once we assume perfect normality, which means that in reality we might be off more often than $5 \%$ of the time.
Is this correct or am I missing something?
 A: 
When we talk about a 95% confidence interval for that sample mean,
  i understand it as saying "given that we assume a normal distribution
  for the sampling distribution of the sample means of samples of size
  n=100, there's a 95% chance that the mean of our sample falls
  at a point under our sampling distribution (normal) curve such that
  the true mean of the population is going to be within 2 standard
  deviations of our sample mean". (is this correct?)

This is not quite correct, and is one of the most common misconceptions in statistics. You can find more discussion here. The 95% confidence interval means that we are 95% confident that the parameter lies in our interval. Another way to think about this is, if we were to repeat the sampling many more times, and create confidence intervals every time, then approximately 95% of those confidence intervals will contain the true parameter.
Otherwise, you are correct that when we appeal to the CLT in this way (in assuming normality of the sample mean), there is an added error since the distribution is never going to be exact. However, our confidence in the estimate might either go up or down. 
Here is a quote from Tsou and Royall(1995)

A popular $95\%$ confidence interval for $E(X)$ based on $n$
  observations is the $t$ interval, $\bar{x} \pm t_{n-1} sn^{-1/2}$,
  where $s^2 = \sum (x - \bar{x}^2)/(n-1)$. This is actually a $95\%$ 
  confidence interval if the $X$'s are iid $N(\theta, \sigma^2)$. But if
  this model is incorrect, then it is no longer true that the coverage
  probability equals the nominal confidence coefficient, .95.

I am going to simulate this behavior in the following R code. I draw samples of size $N = 50$ from first a Normal(5, 1) population and then a $t_{1}$ distribution with mean $\mu = 5$. In the first case, since the population is truly normal, the distribution of the sample mean is exactly normal. In the second case the distribution is a shifted $t_1$ distribution which has much longer tails than the normal distribution. For each of these I simulate a sample of size $N = 50$, 1000 times, make the confidence interval for each time, and check whether $\mu = 5$ is in the interval or not. I return the proportion of time $\mu$ was in the interval, and this number is expected to be .95 if all assumptions hold.
set.seed(100)
## True value of mu
mu <- 5
reps <- 1000 # to demonstrate definition of CI
N <- 50

counting <- vector(length = reps)
## making 95% CI
for(i in 1:reps)
{
    ## When data is really normal and true mean is mu = 5
    ## So CLT hold exactly
    data <-  mu + rnorm(N, mean = 0, sd = 1)
    mu.hat <- mean(data)
    se <- sd(data)/sqrt(N)
    quantile <- qt(.975, df = N-1)
    upper <-  mu.hat + quantile*se
    lower <- mu.hat - quantile*se

    ## Demonstrating how many of the CIs  have mu in them

    counting[i] <- ifelse(upper > mu && lower < mu, 1, 0)
}
mean(counting)
# [1] 0.946

counting <- vector(length = reps)
## making 95% CI
for(i in 1:reps)
{
    ## When data is from t distribution and true mean is still mu = 5
    ## With N = 100 CLT is only approximate
    data <-  mu + rt(N, df = 1)
    mu.hat <- mean(data)
    se <- sd(data)/sqrt(N)
    quantile <- qt(.975, df = N-1)
    upper <-  mu.hat + quantile*se
    lower <- mu.hat - quantile*se

    ## Demonstrating how many of the CIs  have mu in them

    counting[i] <- ifelse(upper > mu && lower < mu, 1, 0)
}
mean(counting)
# [1] 0.986

The first time, I get very close to .95 but the second time I am much higher. So yes, our confidence will be different from .95 if our assumptions don't hold. However if $N$ is large and the data distribution is close to normal, it won't be too far off.
A: The answer by @Greenparker is an excellent answer that highlights a common misconception, but I want to try to address your question more directly. You are correct to say that if your model is wrong (student heights are not normal) then your confidence intervals might not be as accurate as they claim. The issue then lies with identifying an appropriate model and then constructing confidence intervals from that model for the parameters involved (confidence intervals are not something tied solely to models with the normality assumption). 
In some cases, even if your model is wrong, your statistics may be robust to deviations from your model assumptions (e.g. The normal assumption for your heights was wrong, but you end up with 94% confidence intervals rather than 95% confidence intervals...which is just fine). 
And of course as @Greenparker illustrates, the Central Limit theorem is very powerful and holds in many situations. So if your sample size is large enough you can often get away with Normality assumptions. 
All this to say that statistics is quite nuanced, problem specific. 
