When we talk about a 95% confidence interval for that sample mean,
i understand it as saying "given that we assume a normal distribution
for the sampling distribution of the sample means of samples of size
n=100, there's a 95% chance that the mean of our sample falls
at a point under our sampling distribution (normal) curve such that
the true mean of the population is going to be within 2 standard
deviations of our sample mean". (is this correct?)
This is not quite correct, and is one of the most common misconceptions in statistics. You can find more discussion here. The 95% confidence interval means that we are 95% confident that the parameter lies in our interval. Another way to think about this is, if we were to repeat the sampling many more times, and create confidence intervals every time, then approximately 95% of those confidence intervals will contain the true parameter.
Otherwise, you are correct that when we appeal to the CLT in this way (in assuming normality of the sample mean), there is an added error since the distribution is never going to be exact. However, our confidence in the estimate might either go up or down.
Here is a quote from Tsou and Royall(1995)
A popular $95\%$ confidence interval for $E(X)$ based on $n$
observations is the $t$ interval, $\bar{x} \pm t_{n-1} sn^{-1/2}$,
where $s^2 = \sum (x - \bar{x}^2)/(n-1)$. This is actually a $95\%$
confidence interval if the $X$'s are iid $N(\theta, \sigma^2)$. But if
this model is incorrect, then it is no longer true that the coverage
probability equals the nominal confidence coefficient, .95.
I am going to simulate this behavior in the following R code. I draw samples of size $N = 50$ from first a Normal(5, 1) population and then a $t_{1}$ distribution with mean $\mu = 5$. In the first case, since the population is truly normal, the distribution of the sample mean is exactly normal. In the second case the distribution is a shifted $t_1$ distribution which has much longer tails than the normal distribution. For each of these I simulate a sample of size $N = 50$, 1000 times, make the confidence interval for each time, and check whether $\mu = 5$ is in the interval or not. I return the proportion of time $\mu$ was in the interval, and this number is expected to be .95 if all assumptions hold.
set.seed(100)
## True value of mu
mu <- 5
reps <- 1000 # to demonstrate definition of CI
N <- 50
counting <- vector(length = reps)
## making 95% CI
for(i in 1:reps)
{
## When data is really normal and true mean is mu = 5
## So CLT hold exactly
data <- mu + rnorm(N, mean = 0, sd = 1)
mu.hat <- mean(data)
se <- sd(data)/sqrt(N)
quantile <- qt(.975, df = N-1)
upper <- mu.hat + quantile*se
lower <- mu.hat - quantile*se
## Demonstrating how many of the CIs have mu in them
counting[i] <- ifelse(upper > mu && lower < mu, 1, 0)
}
mean(counting)
# [1] 0.946
counting <- vector(length = reps)
## making 95% CI
for(i in 1:reps)
{
## When data is from t distribution and true mean is still mu = 5
## With N = 100 CLT is only approximate
data <- mu + rt(N, df = 1)
mu.hat <- mean(data)
se <- sd(data)/sqrt(N)
quantile <- qt(.975, df = N-1)
upper <- mu.hat + quantile*se
lower <- mu.hat - quantile*se
## Demonstrating how many of the CIs have mu in them
counting[i] <- ifelse(upper > mu && lower < mu, 1, 0)
}
mean(counting)
# [1] 0.986
The first time, I get very close to .95 but the second time I am much higher. So yes, our confidence will be different from .95 if our assumptions don't hold. However if $N$ is large and the data distribution is close to normal, it won't be too far off.
