There has been a lot of words written in the literature about the meaning of a confidence interval and I have been scratching my head so many times about it. Whenever you think you get it, then you read someone else's definition and you are lost again. In the end, the best way for me was to simulate the meaning of a confidence interval (see figure below). And this is the best way since the confidence interval is a frequentist, i.e. a long run frequency, concept, which relies on assumptions for it to work the way it is intended.
In my own words, a confidence interval with a given probability, say 95%, will tell you that you can be 95% certain that this specific interval has coverage of the true population mean and in 5% of the time it does not cover the true population mean. So the probability statement is about the interval, i.e. the interval varies and the population parameter is fixed, or in other words, the interval jumps around the fixed population parameter in a predictiable fashion under repeated sampling (provided that the sampling process occurs at random and under the same conditions).
I try to avoid saying things like there is 95% probability that the true population mean is in that interval because such a statement usually implies that the population parameter is variable and the data is fixed (aka a Bayesian credible interval). However, for any given confidence interval the population parameter is either in the interval or it is not in the interval.
In my mind, the confusion arises since we generally assume that the population parameter is fixed. Then in this case saying that I can be 95% certain that this specific interval has coverage of the true population mean versus there is 95% probability that the true population mean is in that interval has somewhat the same meaning. Once I figured this out for myself, I made sure to explicitly attach the probability statement to the interval whenever talking about a confidence interval.
Here's the simulation (test your own definitions against this figure):
And the code to generate it (may not be the prettiest or the simplest piece of coding but it works):
# required package
library(tidyverse)
library(patchwork)
# create a simulation function for 95% confidence intervals and p-values
simulation <- function(n, mu, stdev) {
s <- rnorm(n, mu, stdev)
tibble(
N = length(s),
sample_mean = mean(s),
sample_sd = sd(s),
sample_se = sample_sd / sqrt(N),
confint_95 = sample_se * qt(0.975, N - 1),
t_statistic = (sample_mean - mu) / sample_se,
p_value = (1 - pt(abs(t_statistic), N - 1)) * 2
)
}
# specify parameters and sample size for each draw or experiment
n <- 10 # sample size
mu <- 0 # population mean
stdev <- 5 # population standard deviation
############ SIMULATION ##############
# set.seed to generate the exact same output
# remove it rerun it to see the stochastic nature of the process
set.seed(126)
# choose number of repetitions
sim <- 1:100
# rerun experiment under exact same conditions
map(sim, ~ simulation(n, mu, stdev)) %>%
bind_rows() %>%
mutate(experiment_id = 1:length(sim)) -> draws
# intervals that do not capture the true population mean are colored red (same as significant p-values)
draws %>%
mutate(
color_confint = if_else(
sample_mean - confint_95 > mu |
sample_mean + confint_95 < mu,
"red",
"black"
),
color_p = if_else(p_value <= 0.05, "red", "black")
) -> draws_colored
# check 95% confidence
draws_colored %>%
group_by(color_confint) %>%
tally() %>%
mutate(prop = n / sum(n) * 100)
# plot 95% confidence intervals
ggplot(data = draws_colored,
aes(x = sample_mean, y = experiment_id, color = color_confint)) + geom_point() +
geom_errorbarh(aes(xmax = sample_mean + confint_95, xmin = sample_mean - confint_95)) +
geom_vline(xintercept = mu) +
scale_colour_manual(name = "95% CI", values = c("black", "red")) +
labs(x = "Sample means", y = "Experiment ID") + theme(legend.position = "none") -> p1
# plot p-values
ggplot(data = draws_colored, aes(x = p_value, y = experiment_id, color = color_p)) + geom_point() +
geom_vline(xintercept = 0.05) +
scale_colour_manual(name = "p-values", values = c("black", "red")) +
labs(x = "p-values", y = "Experiment ID") + theme(legend.position = "none") -> p2
# RERUN CODE HEADER "SIMULATION"
# OUTPUT WILL ALWAYS SLIGHLTY CHANGE SINCE NEW SAMPLES ARE BEING DRAWN RANDOMLY
# ON AVERGAE HOWEVER 5 OUT 100 CONFIDENCE INTERVALS WILL NOT CAPTURE THE TRUE POPULATION MEAN
# -> THE DEFINITION OF A 95% CONFIDENCE INTERVAL
# patch plots together (requires the patchwork package)
p <- p1 / p2
p
