# Computing the confidence interval for two samples but getting slightly different answers

Consider two samples $$X_1,..,X_k$$ and $$Y_1,..,Y_m$$ where $$X_i \sim \mathcal{N}(\mu_x,\,\sigma^{2})\,$$ and $$Y_i \sim \mathcal{N}(\mu_y,\,\sigma^{2})\,.$$ Say $$k=m=100$$ and $$k+m=n$$. Say that the estimated variance and means are $$\hat{\sigma}^{2}=11300,\hat{\mu_x}=16215,\hat{\mu_y}=15669$$

I am interested in the $$95$$ percent confidence interval of the difference of the means. This is given as:

$$16215-15669\pm t_{df=198,1-\frac{\alpha }{2}=0.975}\cdot\sqrt{11300}\cdot \sqrt{\frac{1}{100}+\frac{1}{100}}=(575.64,516.35)$$

I wanna know why I don't get the same interval but a slightly different one when when I consider $$X_i-Y_i \sim \mathcal{N}(16215-15669,\,2 \sigma^{2})\,.$$ and try to build a confidence interval from this and not $$X$$ and $$Y$$ seperated. The random variable, $$X_i-Y_i$$ have now $$100$$ observations. The $$95$$ percent confidence interval will be:

$$16215-15669\pm t_{df=99,1-\frac{\alpha }{2}=0.975}\cdot\frac{\sqrt{2 \cdot 11300}}{\sqrt{100}}=(575.82, 516.17)$$

Can someone help me understand the reason of the difference?

• It could be that the order of operation creates different rounding error. What method & what level of precision did you use to do these calculations? – Michael R. Chernick Oct 14 '19 at 19:06
• The difference is due to the different degrees of freedom of the T-distribution. Notice that, except for the degrees of freedom, the upper and lower confidence intervals are equal even though they are written in different ways. X and Y have 200 observations in total so the degrees of freedom is 198 in the upper CI, and it is 99 in the lower one because X-Y have 100 observations. So it is not rounding errors or because of precision. Intuitively they should be completely equal, but they are not, and I don't understand how. – OBIEK Oct 14 '19 at 20:30
• I used the same method for both. mean.estimate + quantile of T_distribution(df=n-1) * SE(mean.estimator)/sqrt(n) – OBIEK Oct 14 '19 at 20:37
• I didn't notice the difference in degrees of freedom 198 vs 99 but did you look at how close the two t distributions are to each other & the normal distributions? they should be very close to each other. – Michael R. Chernick Oct 14 '19 at 21:40

I will suppose that $$X_1,\ldots,X_k$$ is i.i.d., $$Y_1,\ldots,Y_m$$ i.i.d. and $$X_i$$ is independent of $$Y_j$$ for every i,j. There is no reason for the intervals to be the same. They are different approaches to create a confidence interval for $$\Delta = \mu_X-\mu_Y$$.

Notice that you used the same estimator for the variance in both cases, but they are actually different in each case:

$$\textbf{(i)}$$ In the first case, the estimator for the variance is the pooled variance:

$$S_p^2 = \frac{(k-1)S_x + (m-1)S_y}{k+m-1} \overset{k = m}{=} \frac{S_x+S_y}{2}\quad,$$

where

$$S_x^2 = \frac{1}{k-1}\sum_{i=1}^k(X_i-\bar{X})^2$$

and

$$S_y^2 = \frac{1}{m-1}\sum_{i=1}^m(Y_i-\bar{Y})^2\quad.$$

Taking k=m=100, we will have that $$\frac{\bar{X}-\bar{Y}}{\sqrt{S_p^2}\sqrt{\frac{1}{100} + \frac{1}{100}}} \sim t_{198;0.975}$$.

$$\textbf{(ii)}$$ In the second case, that only exists when k=m, the variance estimator is the sample variance for the difference

$$2\hat{\sigma}^2 = \frac{1}{k-1}\sum_{i=1}^k( (X_i-Y_i) - (\bar{X} - \bar{Y}) )^2 \overset{algebra}{=} 2S_p^2 - \frac{2k}{k-1}\widehat{Cov}(X,Y)\quad,$$

where

$$\widehat{Cov}(X,Y) = \frac{1}{k}\sum_{i=1}^k(X_i-\bar{X})(Y_i-\bar{Y})\quad.$$

Then we will have that $$\frac{\bar{X}-\bar{Y}}{\sqrt{\frac{2\hat{\sigma}^2}{100}}} \sim t_{99;0.975}$$.

The degrees of freedom are also different: In the first case you are estimating three quantities: $$\mu_x,\mu_y$$ and $$\sigma^2$$; in the second one you are estimating two quantities: $$\Delta = \mu_x - \mu_y$$ and $$\sigma^2$$.

Intuitively, I would expect the first procedure to be the best one in the sense of producing smaller lengths confidence intervals for a fixed coefficient. I think this is the case because, when passing from the first setting for the second one, you are "losing information" about the correlation of $$X$$ and $$Y$$. But again, I do not have a mathematical proof for this assertion.

I did some simulations and it appears to be so in most cases, but there is a small probability of the second method producing a confidence interval of smaller length.

• (+1) for showing the role of covariance. – BruceET Oct 15 '19 at 0:17
• Thanks for the comment. So you say that when X_i and Y_j are independent, X_i iid and Y_j iid for all (i,j) and we have unpaired dataset with equal sample sizes, then we can use both approaches even though they give slightly different answers (the difference being due to the different degrees of freedom). And this can be explained in the number of estimators we are estimating? Should we then expect the difference being larger for lower samples because the estimators would have higher variances? – OBIEK Oct 16 '19 at 16:40
• Yea the difference is getting larger for smaller sample sizes. But is it still acceptable to use both approaches for small sample sizes? – OBIEK Oct 16 '19 at 16:45
• The difference is due to how to the information is being used in the construction of the CI. There are differences in the estimation of the variance and in the degrees of freedom, which is related to the number of parameters estimated. Both approaches are correct because their assumptions are met, but the first one should produce better results because it uses more information. I will write another comment to give an example. – Lucas Prates Oct 16 '19 at 23:28
• Imagine the following: you calculate $Z_i = X_i - Y_i$ and send it to a friend, giving him only the values of $Z_i$, telling that they are i.i.d. from a $\mathcal{N}(\Delta, 2\sigma^2)$ distribution. Your friend can estimate $\Delta$ and $\sigma^2$, but he cannot estimate $\mu_X$ and $\mu_Y$. He can too construct a CI for $\Delta$ using the second approach, he just constructs a CI for the mean of the $Z_i$. All the assumptions hold, so he is correct to do so. But notice that he has less information than you, therefore he should have worst results. – Lucas Prates Oct 16 '19 at 23:35

Two independent samples. Your first method is correct. Specifically, suppose $$n_1 = 20, \mu_1 = 50, \sigma_1 = 3$$ and $$n_2 = 30, \mu_2 = 55, \sigma_2 = 3.$$ Note the equal standard deviations.

Using R to simulate data to these specifications, we have the following:

set.seed(1410)
x = rnorm(20, 50, 3);  y = rnorm(30, 55, 3)
summary(x);  sd(x)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
40.06   47.89   50.52   49.92   52.23   56.82
 3.821217  # SD
summary(y);  sd(y)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
48.27   52.85   54.48   54.59   56.56   60.07
 3.138052  # SD


Then the pooled version of the two-sample t test, t.test with parameter var.eq=T, gives a 95% CI $$(-6.654 -2.679)$$ for $$\mu_x - \mu_y,$$ using the pooled estimate $$S_p^2 = \frac{19S_x^2 + 29S_y^2}{48}$$ to estimate $$\sigma^2.$$ [The T statistic is -4.72.]

t.test(x, y, var.eq=T)$conf.int  -6.654289 -2.678636 attr(,"conf.level")  0.95  Are samples independent or paired? Your second method does not make sense: If the two sample sizes are unequal $$(n_1 = 20 \ne n_2 = 30),$$ as in my data, then it isn't clear how to interpret $$X_I = Y_i.$$ If the two sample sizes are equal, and data are paired, so that differences $$D_i = X_i - Y_i$$ make sense, then you are correct that $$\mu_D = \mu_x - \mu_y.$$ Computing the variance $$\sigma_D^2$$ requires knowledge of the covariance between the two variables. (Your result is OK if the X-sample and the Y-sample are independent.) Paired data might be viral blood counts of $$n$$ subjects before and after taking an anti-viral drug. For such paired data, you might use a paired t test to say whether the drug has a statistically significant effect on viral 'loads' of patients. In that case, the paired t test is equivalent to a one-sample t test on the $$D_i.$$ This test would use $$\bar D$$ to estimate $$\mu_x-\mu_y,$$ $$S_D^2$$ to estimate $$\sigma_D^2,$$ and the test statistic $$T = \frac{\bar D}{S_D/\sqrt{n}}.$$ For $$n=50$$ subjects in such a preliminary clinical trial, you might have differences $$D_i$$ and use a t test as shown below. Because the 95% CI $$(-2.37,\, -1.78)$$ does not include $$0,$$ one can conclude that the drug has a statistically significant effect. [Doctors should ponder whether the observed difference of about $$-2$$ (on whatever measurement scale is being used to measure viral load) is of practical importance.] d = rnorm(50, -2, 1) summary(d); sd(d) Min. 1st Qu. Median Mean 3rd Qu. Max. -4.263 -2.716 -1.943 -2.075 -1.286 0.636  1.035901 t.test(d)$conf.int

One Sample t-test

data:  d
t = -14.166, df = 49, p-value < 2.2e-16
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
-2.369734 -1.780934
sample estimates:
mean of x
-2.075334


Note: The data summaries provide the necessary information to find the test statistics for the two t tests shown. It might be worthwhile for you to compute test statistics and confidence intervals with a calculator to see if your results match results shown in R.

Here are summaries of $$X_i, Y_i$$ that might have led to the $$D_i$$ in my paired example:

summary(x1); sd(x1)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
22.93   38.29   47.47   46.75   54.78   68.04
 11.06152
summary(x2); sd(x2)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
25.71   41.62   49.23   48.82   56.97   69.78
 10.8998
summary(x1-x2);  sd(x1-x2)          # same as d
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
-4.263  -2.716  -1.943  -2.075  -1.286   0.636
 1.035901


Inappropriate pooled 2-sample t test. Note use of the parameter var.eq=T to get the pooled test (assuming variances of $$X_1$$ and $$X_2$$ are equal). Having ignored pairing, this test does not give a significant result.

t.test(x1, x2, var.eq=T)

Two Sample t-test

data:  x1 and x2
t = -0.94497, df = 98, p-value = 0.347
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-6.433605  2.282937
sample estimates:
mean of x mean of y
46.74926  48.82460


Appropriate paired test: Now notice that the hightly significant paired t test gives essentially the same output as a one-sample t test on the differences $$D_i.$$

t.test(x1, x2, pair=T)

Paired t-test

data:  x1 and x2
t = -14.166, df = 49, p-value < 2.2e-16
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-2.369734 -1.780934
sample estimates:
mean of the differences
-2.075334

sample estimates:
mean of x mean of y
46.74926  48.82460


The nice mathematical Answer of @LucasPrates (+1) shows the importance of the covariance between $$X_1$$ and $$X_2$$ in this discussion. The covariance in the numerator of the correlation. In paired data, there is often a positive correlation between $$X_1$$ and $$X_2.$$ If you have two independent samples (of equal size) the correlation between their (unsorted) observations should be essentially $$0.$$ For my fake data $$r = 0.9957.$$

cor(x1,x2)
 0.9956583


A scatterplot of these two samples illustrates the correlation (linear association). That almost all of the points lie on one side of the $$45^o$$ line suggests that the paired test result may show a highly significant result. • Thank you. The experiment is to find out if women talk more than men. So Xi is the number of words a woman talks on a day, and Yi is the number of words a man talks on a day. I can see that the data is non-paired, and it doesn't make sense to say Xi-Yi. But now a new question arises. You say that when we have paired data, we can use one-sample t-test (method 2 in my post). Can we use two-sample t.test (method 1 in my post) when we have paired t-test? For example the blood count case? – OBIEK Oct 14 '19 at 22:59
• I think not because X_i will be dependent on Y_i. Am I right? – OBIEK Oct 14 '19 at 23:12
• No. 2-sample t test would lose important info provided by pairing. Will show that in Addendum. // As for men & women talking, I personally know several of each who chatter incessantly. They'd be outliers. If you have small samples and/or many 'far' outliers, a 2-sample t test may not be appropriate. If variances differ btw men and women, might be best to use Welch version of 2-sample t test. // YES to 2nd comment. – BruceET Oct 14 '19 at 23:14
• Thank you for the help! – OBIEK Oct 14 '19 at 23:34