Hypothesis testing with two independent samples test statistics

Question

I'm learning a bit about hypothesis testing with two independent samples (continuous outcome), and I'm just curious about where some of the equations for the test statistics are derived and/or what they're measuring.

Suppose our first sample is of size $n_1$, has mean $\bar{X_1}$, and standard deviation $\sigma_1$, and analogously, our second sample is of size $n_2$ with mean $\bar{X_2}$, and standard deviation $\sigma_2$. We want to test the null hypothesis $H_0: \bar{X_1} = \bar{X_2}$ and assuming our sample sizes are large enough, we can use the test statistic:

$$z = \dfrac{\bar{X_1}-\bar{X_2}}{S_p\,\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}$$

where $S_p$ is the pooled estimate of the common standard deviation:

$$S_p = \sqrt{\dfrac{(n_1-1)\,\sigma_1^2+(n_2-1)\sigma_2^2}{n_1+n_2-2}}$$

For this example, let's suppose the alternative hypothesis is the first mean is larger than the second i.e. $H_1:\bar{X_1}>\bar{X_2}$.

I'm trying to get an intuitive feel for what the above test statistic $z$ is measuring and where the equation comes from and what the meaning of "pooled estimate of the common standard deviation" means...

Answer 1

The numerator of the test statistic is the difference in sample means (representing an estimated difference in population means)

But in order to judge whether that's more than would reasonably be seen if the null were true, we need some idea of scale. So we want to divide by the standard deviation of the difference in means.

So the denominator is an estimate of the standard deviation of distribution of the difference in means. If the assumptions of the t-test hold or hold nearly enough, it will be a good estimate of that standard deviation.

So the test statistic is a standardized difference in means; it's a kind of "internally-standardized z-score" for the difference, which - because we had to estimate the variance term - will have a t-distribution.

The test assumes equal variance in the two groups. $S_p^2$ is (in a particular sense) our best estimate of that common variance, $\sigma^2$. We have two estimates of the same variance, $\sigma^2$. These are $s_1^2$ and $s_2^2$. We'd like to "average" them in some way to get a good estimate of $\sigma^2$. But an estimate derived from a larger sample is more precise -- it should get more weight in the average; the right weight to use (the one that minimizes the variance of the estimate of $\sigma^2$) weights by the degrees of freedom $\text{df}_i=n_i-1$ (they each lose a degree of freedom in estimating the sample mean). So we estimate $\sigma^2$ by a weighted average, $S_p^2 = w_1 s_1^2 + w_2 s_2^2$, where $w_1 = \frac{\text{df}_1}{\text{df}_1+\text{df}_2}$ and $w_2=\frac{\text{df}_2}{\text{df}_1+\text{df}_2}$. Note that the weights add to $1$. This gives the formula for $S_p^2$:

$$S_p^2 = \frac{\text{df}_1}{\text{df}_1+\text{df}_2} s_1^2 + \frac{\text{df}_2}{\text{df}_1+\text{df}_2} s_2^2 = \frac{n_1-1}{n_1-1+n_2-1} s_1^2 + \frac{n_2-1}{n_1-1+n_2-1} s_2^2\\ = \frac{(n_1-1)s_1^2+(n_2-1)s_2^2 }{n_1-1+n_2-1}= \frac{(n_1-1)s_1^2+(n_2-1)s_2^2 }{n_1+n_2-2}$$

If we knew $\sigma$ the variance of $\bar{X_1}$ would be $\sigma^2/n_1$ and the variance of $\bar{X_2}$ would be $\sigma^2/n_2$. Because we assume the observations in the two samples are independent of each other, the variance of the difference $\bar{X_1}-\bar{X_2}$ is the sum of their variances.

So the variance of $\bar{X_1}-\bar{X_2}$ is $\frac{\sigma^2}{n_1} + \frac{\sigma^2}{n_2}=\sigma^2(\frac{1}{n_1} + \frac{1}{n_2})$.

Consequently its standard deviation (the standard error of the difference) is $\sigma\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}$. Once we replace $\sigma$ by our estimate for it, $S_p$, you have the denominator of the t-statistic.

This is pretty much how t-statistics work in general -- they're estimates of some (raw) effect in the numerator, standardized by an estimate of the standard deviation of that effect in the denominator. If a number of conditions hold (generally a consequence of the assumptions for the test and the null being true) then the test statistic will have a t-distribution, with degrees of freedom coming from the d.f. of the estimate in the denominator.