# Hypothesis testing with two independent samples test statistics

I'm learning a bit about hypothesis testing with two independent samples (continuous outcome), and I'm just curious about where some of the equations for the test statistics are derived and/or what they're measuring.

Suppose our first sample is of size $n_1$, has mean $\bar{X_1}$, and standard deviation $\sigma_1$, and analogously, our second sample is of size $n_2$ with mean $\bar{X_2}$, and standard deviation $\sigma_2$. We want to test the null hypothesis $H_0: \bar{X_1} = \bar{X_2}$ and assuming our sample sizes are large enough, we can use the test statistic:

$$z = \dfrac{\bar{X_1}-\bar{X_2}}{S_p\,\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}$$

where $S_p$ is the pooled estimate of the common standard deviation:

$$S_p = \sqrt{\dfrac{(n_1-1)\,\sigma_1^2+(n_2-1)\sigma_2^2}{n_1+n_2-2}}$$

For this example, let's suppose the alternative hypothesis is the first mean is larger than the second i.e. $H_1:\bar{X_1}>\bar{X_2}$.

I'm trying to get an intuitive feel for what the above test statistic $z$ is measuring and where the equation comes from and what the meaning of "pooled estimate of the common standard deviation" means...

• I'm puzzled by "assuming the sample sizes are large enough". If your sample sizes are large enough to compute an $S_p$ (the smallest possible being samples of 1 and 2 in either order), then you'll have a perfectly valid $t$-statstic, which will have a $t$-distribution under the null (and given the assumptions of the test) – Glen_b Oct 5 '17 at 22:55
• Ah I mentioned that because I thought $n_1$ and $n_2$ need to be large enough to use a z score because if $n_1$ and $n_2$ are small then you use a $t$ test because then your data will have a $t$ distribution. – clueless_undergrad37 Oct 5 '17 at 23:09
• But the statistic you gave is a t-statistic! Calling the t-statistic "z" doesn't do anything. [... Edit:] Oh, wait, I think I get it -- I guess you might want to invoke Slutsky's theorem for the ratio (assuming the sample sizes are large enough that you could treat the estimate in the denominator as essentially having no error) and then apply the CLT to the numerator. Well, okay but you could need quite large sample sizes before that's going to work well. Doesn't alter the motivation for the form of the statistic, apart from the claims about "best estimates" of various quantities. – Glen_b Oct 5 '17 at 23:54

The numerator of the test statistic is the difference in sample means (representing an estimated difference in population means)

But in order to judge whether that's more than would reasonably be seen if the null were true, we need some idea of scale. So we want to divide by the standard deviation of the difference in means.

So the denominator is an estimate of the standard deviation of distribution of the difference in means. If the assumptions of the t-test hold or hold nearly enough, it will be a good estimate of that standard deviation.

So the test statistic is a standardized difference in means; it's a kind of "internally-standardized z-score" for the difference, which - because we had to estimate the variance term - will have a t-distribution.

The test assumes equal variance in the two groups. $S_p^2$ is (in a particular sense) our best estimate of that common variance, $\sigma^2$. We have two estimates of the same variance, $\sigma^2$. These are $s_1^2$ and $s_2^2$. We'd like to "average" them in some way to get a good estimate of $\sigma^2$. But an estimate derived from a larger sample is more precise -- it should get more weight in the average; the right weight to use (the one that minimizes the variance of the estimate of $\sigma^2$) weights by the degrees of freedom $\text{df}_i=n_i-1$ (they each lose a degree of freedom in estimating the sample mean). So we estimate $\sigma^2$ by a weighted average, $S_p^2 = w_1 s_1^2 + w_2 s_2^2$, where $w_1 = \frac{\text{df}_1}{\text{df}_1+\text{df}_2}$ and $w_2=\frac{\text{df}_2}{\text{df}_1+\text{df}_2}$. Note that the weights add to $1$. This gives the formula for $S_p^2$:

$$S_p^2 = \frac{\text{df}_1}{\text{df}_1+\text{df}_2} s_1^2 + \frac{\text{df}_2}{\text{df}_1+\text{df}_2} s_2^2 = \frac{n_1-1}{n_1-1+n_2-1} s_1^2 + \frac{n_2-1}{n_1-1+n_2-1} s_2^2\\ = \frac{(n_1-1)s_1^2+(n_2-1)s_2^2 }{n_1-1+n_2-1}= \frac{(n_1-1)s_1^2+(n_2-1)s_2^2 }{n_1+n_2-2}$$

If we knew $\sigma$ the variance of $\bar{X_1}$ would be $\sigma^2/n_1$ and the variance of $\bar{X_2}$ would be $\sigma^2/n_2$. Because we assume the observations in the two samples are independent of each other, the variance of the difference $\bar{X_1}-\bar{X_2}$ is the sum of their variances.

So the variance of $\bar{X_1}-\bar{X_2}$ is $\frac{\sigma^2}{n_1} + \frac{\sigma^2}{n_2}=\sigma^2(\frac{1}{n_1} + \frac{1}{n_2})$.

Consequently its standard deviation (the standard error of the difference) is $\sigma\sqrt{\frac{1}{n_1} + \frac{1}{n_2}}$. Once we replace $\sigma$ by our estimate for it, $S_p$, you have the denominator of the t-statistic.

This is pretty much how t-statistics work in general -- they're estimates of some (raw) effect in the numerator, standardized by an estimate of the standard deviation of that effect in the denominator. If a number of conditions hold (generally a consequence of the assumptions for the test and the null being true) then the test statistic will have a t-distribution, with degrees of freedom coming from the d.f. of the estimate in the denominator.