Computing the confidence interval for two samples but getting slightly different answers

Question

Consider two samples $X_1,..,X_k$ and $Y_1,..,Y_m$ where $X_i \sim \mathcal{N}(\mu_x,\,\sigma^{2})\,$ and $Y_i \sim \mathcal{N}(\mu_y,\,\sigma^{2})\,.$ Say $k=m=100$ and $k+m=n$. Say that the estimated variance and means are $\hat{\sigma}^{2}=11300,\hat{\mu_x}=16215,\hat{\mu_y}=15669$

I am interested in the $95$ percent confidence interval of the difference of the means. This is given as:

$16215-15669\pm t_{df=198,1-\frac{\alpha }{2}=0.975}\cdot\sqrt{11300}\cdot \sqrt{\frac{1}{100}+\frac{1}{100}}=(575.64,516.35)$

I wanna know why I don't get the same interval but a slightly different one when when I consider $X_i-Y_i \sim \mathcal{N}(16215-15669,\,2 \sigma^{2})\,.$ and try to build a confidence interval from this and not $X$ and $Y$ seperated. The random variable, $X_i-Y_i$ have now $100$ observations. The $95$ percent confidence interval will be:

$16215-15669\pm t_{df=99,1-\frac{\alpha }{2}=0.975}\cdot\frac{\sqrt{2 \cdot 11300}}{\sqrt{100}}=(575.82, 516.17)$

Can someone help me understand the reason of the difference?

Answer 1

I will suppose that $X_1,\ldots,X_k$ is i.i.d., $Y_1,\ldots,Y_m$ i.i.d. and $X_i$ is independent of $Y_j$ for every i,j. There is no reason for the intervals to be the same. They are different approaches to create a confidence interval for $\Delta = \mu_X-\mu_Y$.

Notice that you used the same estimator for the variance in both cases, but they are actually different in each case:

$\textbf{(i)}$ In the first case, the estimator for the variance is the pooled variance:

$$S_p^2 = \frac{(k-1)S_x + (m-1)S_y}{k+m-1} \overset{k = m}{=} \frac{S_x+S_y}{2}\quad,$$

where

$$S_x^2 = \frac{1}{k-1}\sum_{i=1}^k(X_i-\bar{X})^2$$

and

$$S_y^2 = \frac{1}{m-1}\sum_{i=1}^m(Y_i-\bar{Y})^2\quad.$$

Taking k=m=100, we will have that $\frac{\bar{X}-\bar{Y}}{\sqrt{S_p^2}\sqrt{\frac{1}{100} + \frac{1}{100}}} \sim t_{198;0.975}$.

$\textbf{(ii)}$ In the second case, that only exists when k=m, the variance estimator is the sample variance for the difference

$$2\hat{\sigma}^2 = \frac{1}{k-1}\sum_{i=1}^k( (X_i-Y_i) - (\bar{X} - \bar{Y}) )^2 \overset{algebra}{=} 2S_p^2 - \frac{2k}{k-1}\widehat{Cov}(X,Y)\quad,$$

where

$$\widehat{Cov}(X,Y) = \frac{1}{k}\sum_{i=1}^k(X_i-\bar{X})(Y_i-\bar{Y})\quad.$$

Then we will have that $\frac{\bar{X}-\bar{Y}}{\sqrt{\frac{2\hat{\sigma}^2}{100}}} \sim t_{99;0.975}$.

The degrees of freedom are also different: In the first case you are estimating three quantities: $\mu_x,\mu_y$ and $\sigma^2$; in the second one you are estimating two quantities: $\Delta = \mu_x - \mu_y$ and $\sigma^2$.

Intuitively, I would expect the first procedure to be the best one in the sense of producing smaller lengths confidence intervals for a fixed coefficient. I think this is the case because, when passing from the first setting for the second one, you are "losing information" about the correlation of $X$ and $Y$. But again, I do not have a mathematical proof for this assertion.

I did some simulations and it appears to be so in most cases, but there is a small probability of the second method producing a confidence interval of smaller length.

Answer 2

Two independent samples. Your first method is correct. Specifically, suppose $n_1 = 20, \mu_1 = 50, \sigma_1 = 3$ and $n_2 = 30, \mu_2 = 55, \sigma_2 = 3.$ Note the equal standard deviations.

Using R to simulate data to these specifications, we have the following:

set.seed(1410)
x = rnorm(20, 50, 3);  y = rnorm(30, 55, 3)
summary(x);  sd(x)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
  40.06   47.89   50.52   49.92   52.23   56.82 
[1] 3.821217  # SD
summary(y);  sd(y)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
   48.27   52.85   54.48   54.59   56.56   60.07 
[1] 3.138052  # SD

Then the pooled version of the two-sample t test, t.test with parameter var.eq=T, gives a 95% CI $(-6.654 -2.679)$ for $\mu_x - \mu_y,$ using the pooled estimate $S_p^2 = \frac{19S_x^2 + 29S_y^2}{48}$ to estimate $\sigma^2.$ [The T statistic is -4.72.]

t.test(x, y, var.eq=T)$conf.int
[1] -6.654289 -2.678636
 attr(,"conf.level")
 [1] 0.95

Are samples independent or paired? Your second method does not make sense: If the two sample sizes are unequal $(n_1 = 20 \ne n_2 = 30),$ as in my data, then it isn't clear how to interpret $X_I = Y_i.$

If the two sample sizes are equal, and data are paired, so that differences $D_i = X_i - Y_i$ make sense, then you are correct that $\mu_D = \mu_x - \mu_y.$ Computing the variance $\sigma_D^2$ requires knowledge of the covariance between the two variables. (Your result is OK if the X-sample and the Y-sample are independent.)

Paired data might be viral blood counts of $n$ subjects before and after taking an anti-viral drug. For such paired data, you might use a paired t test to say whether the drug has a statistically significant effect on viral 'loads' of patients. In that case, the paired t test is equivalent to a one-sample t test on the $D_i.$ This test would use $\bar D$ to estimate $\mu_x-\mu_y,$ $S_D^2$ to estimate $\sigma_D^2,$ and the test statistic $T = \frac{\bar D}{S_D/\sqrt{n}}.$

For $n=50$ subjects in such a preliminary clinical trial, you might have differences $D_i$ and use a t test as shown below. Because the 95% CI $(-2.37,\, -1.78)$ does not include $0,$ one can conclude that the drug has a statistically significant effect. [Doctors should ponder whether the observed difference of about $-2$ (on whatever measurement scale is being used to measure viral load) is of practical importance.]

d = rnorm(50, -2, 1)
summary(d);  sd(d)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 -4.263  -2.716  -1.943  -2.075  -1.286   0.636 
[1] 1.035901

t.test(d)$conf.int

           One Sample t-test

data:  d
t = -14.166, df = 49, p-value < 2.2e-16
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 -2.369734 -1.780934
sample estimates:
mean of x 
-2.075334 

Note: The data summaries provide the necessary information to find the test statistics for the two t tests shown. It might be worthwhile for you to compute test statistics and confidence intervals with a calculator to see if your results match results shown in R.

Addendum per Comment.

Here are summaries of $X_i, Y_i$ that might have led to the $D_i$ in my paired example:

summary(x1); sd(x1)
  Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 22.93   38.29   47.47   46.75   54.78   68.04 
[1] 11.06152
summary(x2); sd(x2)
  Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 25.71   41.62   49.23   48.82   56.97   69.78 
[1] 10.8998
summary(x1-x2);  sd(x1-x2)          # same as d
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 -4.263  -2.716  -1.943  -2.075  -1.286   0.636 
[1] 1.035901

Inappropriate pooled 2-sample t test. Note use of the parameter var.eq=T to get the pooled test (assuming variances of $X_1$ and $X_2$ are equal). Having ignored pairing, this test does not give a significant result.

t.test(x1, x2, var.eq=T)

        Two Sample t-test

data:  x1 and x2
t = -0.94497, df = 98, p-value = 0.347
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -6.433605  2.282937
sample estimates:
mean of x mean of y 
 46.74926  48.82460 

Appropriate paired test: Now notice that the hightly significant paired t test gives essentially the same output as a one-sample t test on the differences $D_i.$

t.test(x1, x2, pair=T)

        Paired t-test

data:  x1 and x2
t = -14.166, df = 49, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -2.369734 -1.780934
sample estimates:
mean of the differences 
              -2.075334 

sample estimates:
mean of x mean of y 
 46.74926  48.82460 

The nice mathematical Answer of @LucasPrates (+1) shows the importance of the covariance between $X_1$ and $X_2$ in this discussion. The covariance in the numerator of the correlation. In paired data, there is often a positive correlation between $X_1$ and $X_2.$ If you have two independent samples (of equal size) the correlation between their (unsorted) observations should be essentially $0.$ For my fake data $r = 0.9957.$

cor(x1,x2)
[1] 0.9956583

A scatterplot of these two samples illustrates the correlation (linear association). That almost all of the points lie on one side of the $45^o$ line suggests that the paired test result may show a highly significant result.