What metric does the 95% confidence interval of a 2-sample t-test cover?

Question

I'm trying to better understand what the 95% confidence interval is supposed to cover (in the context of a two-sample t-test). I had always been under the impression that the 95% confidence interval meant that 95% of the time, the confidence interval would contain the true average treatment effect (ATE). However, it has come to my attention recently that the coverage could be for the ATE for the sampled population of users.

In other words, if I have a data generating process as follows:

x = np.random.randint(0,2,1000)
control_values = np.random.normal(0,1,1000)
treatment_values = np.random.normal(1,1,1000)

Yt = treatment_values[x == 1]
Yc = control_values[x == 0]

We compute the 95% confidence interval as follows:

ci_width = np.sqrt(np.var(Yt)/len(Yt) + np.var(Yc)/len(Yc))* 1.96

[(Yt.mean() - Yc.mean()) - ci_width, (Yt.mean() - Yc.mean()) + ci_width]

Should the 95% confidence interval be covering:

1. The true ATE, which is 1, based on the data generating process
2. The ATE of the sampled population ( treatment_values.mean() - control_values.mean())?

Edit: Let me give another example that will hopefully illustrate the issue more clearly.

X = np.random.uniform(-5, 5, 1000) 
Y_Control = np.random.normal(X, 1, 1000)
Y_Treatment = np.random.normal(X + 1, 1, 1000)

Here, X is the pre-experiment variable, and Y is the post-experiment variable. Y_Control is the post-experiment variable observed under Control, and Y_Treatment is the post-experiment variable observed under Treatment.

The ATE of the sample should be np.mean(Y_Treatment - Y_Control); however, for each experiment, assuming a 50:50 split, we only observe half of the values in Y_Control, and half of the values in Y_Treatment. So the the observed difference is only np.mean(Y_Treatment[users_in_treatment]) - np.mean(Y_Control[users_in_control]). Note that this is not equivalent to np.mean(Y_Treatment - Y_Control)

My question is, should the 95% CI computed from the observed values, Y_Treatment[users_in_treatment] and Y_Control[users_in_control] provide the appropriate coverage for the:

1. true ATE of 1 (assuming we had infinite users)
2. the (unobserved) ATE of the sample: np.mean(Y_Treatment - Y_Control)

Answer 1

In general, a confidence interval covers the population parameter. Some procedures do better or worse than others, but when we say “95% confidence” we mean that repeating the procedure over and over would result in asserted confidence intervals containing the true value in 95% of those repeats.

Given that treatment_values.mean() - control_values.mean() is in the center of the confidence intervals you calculate, every confidence interval will contain this value.