2
$\begingroup$

I'm trying to better understand what the 95% confidence interval is supposed to cover (in the context of a two-sample t-test). I had always been under the impression that the 95% confidence interval meant that 95% of the time, the confidence interval would contain the true average treatment effect (ATE). However, it has come to my attention recently that the coverage could be for the ATE for the sampled population of users.

In other words, if I have a data generating process as follows:

x = np.random.randint(0,2,1000)
control_values = np.random.normal(0,1,1000)
treatment_values = np.random.normal(1,1,1000)

Yt = treatment_values[x == 1]
Yc = control_values[x == 0]

We compute the 95% confidence interval as follows:

ci_width = np.sqrt(np.var(Yt)/len(Yt) + np.var(Yc)/len(Yc))* 1.96

[(Yt.mean() - Yc.mean()) - ci_width, (Yt.mean() - Yc.mean()) + ci_width]

Should the 95% confidence interval be covering:

  1. The true ATE, which is 1, based on the data generating process
  2. The ATE of the sampled population ( treatment_values.mean() - control_values.mean())?

Edit: Let me give another example that will hopefully illustrate the issue more clearly.

X = np.random.uniform(-5, 5, 1000) 
Y_Control = np.random.normal(X, 1, 1000)
Y_Treatment = np.random.normal(X + 1, 1, 1000)

Here, X is the pre-experiment variable, and Y is the post-experiment variable. Y_Control is the post-experiment variable observed under Control, and Y_Treatment is the post-experiment variable observed under Treatment.

The ATE of the sample should be np.mean(Y_Treatment - Y_Control); however, for each experiment, assuming a 50:50 split, we only observe half of the values in Y_Control, and half of the values in Y_Treatment. So the the observed difference is only np.mean(Y_Treatment[users_in_treatment]) - np.mean(Y_Control[users_in_control]). Note that this is not equivalent to np.mean(Y_Treatment - Y_Control)

My question is, should the 95% CI computed from the observed values, Y_Treatment[users_in_treatment] and Y_Control[users_in_control] provide the appropriate coverage for the:

  1. true ATE of 1 (assuming we had infinite users)
  2. the (unobserved) ATE of the sample: np.mean(Y_Treatment - Y_Control)
$\endgroup$
3
  • $\begingroup$ Characterization (2) is not a confidence interval -- it would be something else. An interval for the ATE of the sample you give in (2) would be superfluous: you know exactly what it is. It's unclear what distinction you might be making between the "data generating process" and "sampled population" if, by the latter, you don't mean the sample itself. $\endgroup$
    – whuber
    Aug 9, 2022 at 16:29
  • $\begingroup$ Hi, I edited my question, I hope it is more clear now! Would you mind taking another look? $\endgroup$
    – monkeybiz7
    Aug 9, 2022 at 17:40
  • $\begingroup$ The CI is designed to cover the true ATE of 1 with asymptotically valid coverage $\endgroup$
    – Ben
    Aug 9, 2022 at 18:52

1 Answer 1

0
$\begingroup$

In general, a confidence interval covers the population parameter. Some procedures do better or worse than others, but when we say “95% confidence” we mean that repeating the procedure over and over would result in asserted confidence intervals containing the true value in 95% of those repeats.

Given that treatment_values.mean() - control_values.mean() is in the center of the confidence intervals you calculate, every confidence interval will contain this value.

$\endgroup$
1
  • $\begingroup$ So treatment_values.mean() - control_values.mean() is not in the center of the confidence intervals I calculate because for a given XP (assuming a 50:50 split) I only observe half of the values from treatment_values and half of the values from control_values. I think my original question wasn't clear and I've edited to hopefully make it more understandable. Would you mind taking another look? Thank you! $\endgroup$
    – monkeybiz7
    Aug 9, 2022 at 17:42

This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .