Intuition for the standard error of the difference of sample means

Question

I read in Wilcox, 2003 p. 247 that the standard error of the difference between two sample means is (assuming the normality and homoskedasticity assumptions):

$\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}$

Rather than simply adding the two sample standard errors as in:

$\frac{\sigma_1}{\sqrt{n_1}} + \frac{\sigma_2}{\sqrt{n_2}}$

What is the intuition behind taking the square of the sum of the two variances divided into their respective sample size, rather than the sum of the standard errors?

Answer 1

You seem to be thinking that $\sqrt{\text{Var}(\bar X-\bar Y)} = \sqrt{\text{Var}(\bar X)} + \sqrt{\text{Var}(\bar Y)}$.

This is not the case for independent variables.

For $X,Y$ independent, $\text{Var}(\bar X-\bar Y) = \text{Var}(\bar X) + \text{Var}(\bar Y)$

Further,

$\text{Var}(\bar X) = \text{Var}(\frac{1}{n}\sum_iX_i) = \frac{1}{n^2}\text{Var}(\sum_iX_i)= \frac{1}{n^2}\sum_i\text{Var}(X_i)= \frac{1}{n^2}\cdot n\cdot\sigma^2_1= \sigma^2_1/n$

(if the $X_i$ are independent of each other).

http://en.wikipedia.org/wiki/Variance#Basic_properties

In summary: the correct term:

$\color{red}{(1)}$ has $\sigma^2/n$ terms because we're looking at averages and that's the variance of an average of independent random variables;

$\color{red}{(2)}$ has a $+$ because the two samples are independent, so their variances (of the averages) add; and

$\color{red}{(3)}$ has a square root because we want the standard deviation of the distribution of the difference in sample means (the standard error of the difference in means). The part under the bar of the square root is the variance of the difference (the square of the standard error). Taking square roots of squared standard errors gives us standard errors.

The reason why we don't just add standard errors is standard errors don't add - the standard error of the difference in means is NOT the sum of the standard errors of the sample means for independent samples - the sum will always be too large. The variances do add, though, so we can use that to work out the standard errors.

---

Here's some intuition about why it's not standard deviations that add, rather than variances.

To make things a little simpler, just consider adding random variables.

If $Z = X+Y$, why is $\sigma_Z < \sigma_X+\sigma_Y$?

Imagine $Y = kX$ (for $k\neq 0$); that is, $X$ and $Y$ are perfectly linearly dependent. That is, they always 'move together' in the same direction and in proportion.

Then $Z = (k+1)X$ - which is simply a rescaling. Clearly $\sigma_Z = (k+1)\sigma_X = \sigma_X+\sigma_Y$.

That is, when $X$ and $Y$ are perfectly positively linearly dependent, always moving up or down together, standard deviations add.

When they don't always move up or down together, sometimes they move opposite directions. That means that their movements partly 'cancel out', yielding a smaller standard deviation than the direct sum.

Answer 2

Algebraic intuition

The standard error of the mean for $n$ independent observations is $\frac{\sigma}{\sqrt{n}}$ where $\sigma$ is the standard deviation.

So if we have two independent samples we have the standard errors for the means of group 1 and group 2.

$$\sigma_{\mu_1}=\frac{\sigma_1}{\sqrt{n_1}}$$ $$\sigma_{\mu_2}=\frac{\sigma_2}{\sqrt{n_2}}$$

If we square these values we get the variance of the mean:

$$\sigma^2_{\mu_1}=\frac{\sigma^2_1}{n_1}$$ $$\sigma^2_{\mu_2}=\frac{\sigma^2_2}{n_2}$$

The variance of the sum or difference of two independent random variables is the sum of the two variances. Thus,

$$\sigma^2_{\mu_1 - \mu_2} =\sigma^2_{\mu_1} + \sigma^2_{\mu_2} = \frac{\sigma^2_1}{n_1} + \frac{\sigma^2_2}{n_2} $$

So if we want the standard error of the difference we take the square root of the variance:

$$\sigma_{\mu_1 - \mu_2} =\sqrt{\sigma^2_{\mu_1} + \sigma^2_{\mu_2}} = \sqrt{\frac{\sigma^2_1}{n_1} + \frac{\sigma^2_2}{n_2}} $$

So I imagine this is intuitive if the component steps are intuitive. In particular it helps if you find intuitive the idea that the variance of the sum of independent variables is the sum of the variances of the component variables.

Fuzzy Intuition

In terms of more general intuition, if $n_1 = n_2$ and $\sigma=\sigma_1=\sigma_2$ then the standard error of the difference between means will be $\sqrt{2}\sigma_\mu\approx 1.4\times \sigma_\mu$. It makes sense that this value of approximately 1.4 is greater than 1 (i.e., the variance of a variable after adding a constant; i.e., equivalent to one sample t-test) and less than 2 (i.e., the standard deviation of the sum of two perfectly correlated variables (with equal variance) and the standard error implied by the formula you mention: $\frac{\sigma_1}{\sqrt{n_1}} + \frac{\sigma_2}{\sqrt{n_2}}$).

Answer 3

You don't square the sum of the variances, you take the square root of the sum of the variances. You do this for the same reason that the standard deviation is the square root of the variance: It make the units the same as the original ones, rather than squared units.

Although we often lose sight of it while doing statistics, the square of a measure involve squaring the measure as well as the number of units. For example, the square of 2 meters is not 4 meters, it is 4 meters squared, more commonly called 4 square meters. The same thing happens with other units that we aren't used to thinking of in this way: e.g. if you are measuring IQ, the square of an IQ is not an IQ of 10,000; it is a squared IQ of 10,000.

You divide by the sample size as a scaling technique. Variances (tend to) go up with sample size; you divide by $n$ to deal with that.