6
$\begingroup$

I read in Wilcox, 2003 p. 247 that the standard error of the difference between two sample means is (assuming the normality and homoskedasticity assumptions):

$\sqrt{\frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2}}$

Rather than simply adding the two sample standard errors as in:

$\frac{\sigma_1}{\sqrt{n_1}} + \frac{\sigma_2}{\sqrt{n_2}}$

What is the intuition behind taking the square of the sum of the two variances divided into their respective sample size, rather than the sum of the standard errors?

$\endgroup$
  • $\begingroup$ If you want standard errors, your second formula shouldn't have squared terms, and it's not the standard error of the distributions but the standard error of the sample means you're talking about there. $\endgroup$ – Glen_b Oct 14 '13 at 23:37
  • $\begingroup$ Yes sample means, not distribution means. Fixed the question. $\endgroup$ – Robert Kubrick Oct 15 '13 at 0:21
  • $\begingroup$ You still have the first error I mentioned. I will fix it for you. $\endgroup$ – Glen_b Oct 15 '13 at 1:22
  • 2
    $\begingroup$ Perhaps the most famous--and likely the earliest--completely intuitive demonstration of this relation was obtained by Galton with his quincunx. (Don't visit Wikipedia or any of the top hits on this term, because they explain only its most trivial uses: the link I gave is the first I could find that reproduces one of Galton's illustrations and hints at the revelations this machine makes possible.) $\endgroup$ – whuber Oct 15 '13 at 5:15
  • 2
    $\begingroup$ The intuition is one of cancellation of independent errors. It took scientists rather a long time--approximately 150 years from the beginning of probability theory in the mid 17th century to the work of Gauss and Laplace around 1800 as applied to combining astronomical observations--to realize that this actually happens with measurements! Because such cancellation is at the heart of the Central Limit Theorem, intuitions for that are relevant here, too. $\endgroup$ – whuber Oct 15 '13 at 5:21
8
$\begingroup$

You seem to be thinking that $\sqrt{\text{Var}(\bar X-\bar Y)} = \sqrt{\text{Var}(\bar X)} + \sqrt{\text{Var}(\bar Y)}$.

This is not the case for independent variables.

For $X,Y$ independent, $\text{Var}(\bar X-\bar Y) = \text{Var}(\bar X) + \text{Var}(\bar Y)$

Further,

$\text{Var}(\bar X) = \text{Var}(\frac{1}{n}\sum_iX_i) = \frac{1}{n^2}\text{Var}(\sum_iX_i)= \frac{1}{n^2}\sum_i\text{Var}(X_i)= \frac{1}{n^2}\cdot n\cdot\sigma^2_1= \sigma^2_1/n$

(if the $X_i$ are independent of each other).

http://en.wikipedia.org/wiki/Variance#Basic_properties

In summary: the correct term:

annotated se formula

$\color{red}{(1)}$ has $\sigma^2/n$ terms because we're looking at averages and that's the variance of an average of independent random variables;

$\color{red}{(2)}$ has a $+$ because the two samples are independent, so their variances (of the averages) add; and

$\color{red}{(3)}$ has a square root because we want the standard deviation of the distribution of the difference in sample means (the standard error of the difference in means). The part under the bar of the square root is the variance of the difference (the square of the standard error). Taking square roots of squared standard errors gives us standard errors.

The reason why we don't just add standard errors is standard errors don't add - the standard error of the difference in means is NOT the sum of the standard errors of the sample means for independent samples - the sum will always be too large. The variances do add, though, so we can use that to work out the standard errors.


Here's some intuition about why it's not standard deviations that add, rather than variances.

To make things a little simpler, just consider adding random variables.

If $Z = X+Y$, why is $\sigma_Z < \sigma_X+\sigma_Y$?

Imagine $Y = kX$ (for $k\neq 0$); that is, $X$ and $Y$ are perfectly linearly dependent. That is, they always 'move together' in the same direction and in proportion.

Then $Z = (k+1)X$ - which is simply a rescaling. Clearly $\sigma_Z = (k+1)\sigma_X = \sigma_X+\sigma_Y$.

That is, when $X$ and $Y$ are perfectly positively linearly dependent, always moving up or down together, standard deviations add.

When they don't always move up or down together, sometimes they move opposite directions. That means that their movements partly 'cancel out', yielding a smaller standard deviation than the direct sum.

$\endgroup$
  • $\begingroup$ I'm ok with the variance algebra, what I am asking is why the square root goes to the sum of the two variances divided into the sample sizes. Notice that in the second (wrong) formula the variance is already squared, that is we use the standard deviation. $\endgroup$ – Robert Kubrick Oct 15 '13 at 1:48
  • 1
    $\begingroup$ Every part of the correct formula is covered by my discussion above. Can you more clearly explain what you don't get and I can point to where it's already covered by my answer. I can try expanding my explanation further. $\endgroup$ – Glen_b Oct 15 '13 at 2:17
  • $\begingroup$ I have gone into more detail. If that doesn't suffice, please indicate clearly where the problem lies. $\endgroup$ – Glen_b Oct 15 '13 at 2:47
  • $\begingroup$ The intuition I can't capture is when you say "...the sum will always be too large". I need to read through @whuber links, if you can elaborate a bit more it would help. Thank you. $\endgroup$ – Robert Kubrick Oct 17 '13 at 12:30
  • $\begingroup$ Do you see that $\sqrt{a^2+b^2}\leq a+b$ (with $a, b\geq0$), and you only have equality if one of them is $0$? (This is just that the hypotenuse of a right angled triangle is smaller than the sum of the other two sides) $\endgroup$ – Glen_b Oct 17 '13 at 20:04
3
$\begingroup$

Algebraic intuition

The standard error of the mean for $n$ independent observations is $\frac{\sigma}{\sqrt{n}}$ where $\sigma$ is the standard deviation.

So if we have two independent samples we have the standard errors for the means of group 1 and group 2.

$$\sigma_{\mu_1}=\frac{\sigma_1}{\sqrt{n_1}}$$ $$\sigma_{\mu_2}=\frac{\sigma_2}{\sqrt{n_2}}$$

If we square these values we get the variance of the mean:

$$\sigma^2_{\mu_1}=\frac{\sigma^2_1}{n_1}$$ $$\sigma^2_{\mu_2}=\frac{\sigma^2_2}{n_2}$$

The variance of the sum or difference of two independent random variables is the sum of the two variances. Thus,

$$\sigma^2_{\mu_1 - \mu_2} =\sigma^2_{\mu_1} + \sigma^2_{\mu_2} = \frac{\sigma^2_1}{n_1} + \frac{\sigma^2_2}{n_2} $$

So if we want the standard error of the difference we take the square root of the variance:

$$\sigma_{\mu_1 - \mu_2} =\sqrt{\sigma^2_{\mu_1} + \sigma^2_{\mu_2}} = \sqrt{\frac{\sigma^2_1}{n_1} + \frac{\sigma^2_2}{n_2}} $$

So I imagine this is intuitive if the component steps are intuitive. In particular it helps if you find intuitive the idea that the variance of the sum of independent variables is the sum of the variances of the component variables.

Fuzzy Intuition

In terms of more general intuition, if $n_1 = n_2$ and $\sigma=\sigma_1=\sigma_2$ then the standard error of the difference between means will be $\sqrt{2}\sigma_\mu\approx 1.4\times \sigma_\mu$. It makes sense that this value of approximately 1.4 is greater than 1 (i.e., the variance of a variable after adding a constant; i.e., equivalent to one sample t-test) and less than 2 (i.e., the standard deviation of the sum of two perfectly correlated variables (with equal variance) and the standard error implied by the formula you mention: $\frac{\sigma_1}{\sqrt{n_1}} + \frac{\sigma_2}{\sqrt{n_2}}$).

$\endgroup$
2
$\begingroup$

You don't square the sum of the variances, you take the square root of the sum of the variances. You do this for the same reason that the standard deviation is the square root of the variance: It make the units the same as the original ones, rather than squared units.

Although we often lose sight of it while doing statistics, the square of a measure involve squaring the measure as well as the number of units. For example, the square of 2 meters is not 4 meters, it is 4 meters squared, more commonly called 4 square meters. The same thing happens with other units that we aren't used to thinking of in this way: e.g. if you are measuring IQ, the square of an IQ is not an IQ of 10,000; it is a squared IQ of 10,000.

You divide by the sample size as a scaling technique. Variances (tend to) go up with sample size; you divide by $n$ to deal with that.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.