How does the standard error work?

Question

I have been looking into the inner-workings of the standard error recently, and I found myself unable to understand how it works. My understanding of the standard error is that it is the standard deviation of the distribution of sample means. My questions are:

• how do we know the standard error is the standard deviation of the sample means when we usually take just a single sample?

• why doesnt the equation to calculate the standard error mirror the standard deviation equation for a single sample?

Answer 1

Yes, the standard error of the mean (SEM) is the standard deviation (SD) of the means. (Standard error is another way to say SD of a sampling distribution. In this case, the sampling distribution is means for samples of a fixed size, say N.) There is a mathematical relationship between the SEM and the population SD: SEM = population SD / the square root of N. This mathematical relationship is very helpful, since we almost never have a direct estimate of the SEM but we do have an estimate of the population SD (namely the SD of our sample). As to your second question, if you were to collect multiple samples of size N and calculate the mean for each sample you could estimate the SEM simply by calculating the SD of the means. So the formula for SEM does indeed mirror the formula for the SD of a single sample.

Answer 2

Suppose $X_1, X_2, \ldots, X_n$ are independent and identically distributed. This is the situation I am pretty sure you are referring to. Let their common mean be $\mu$ and their common variance be $\sigma^2$.

Now the sample mean is $X_b=\sum_i X_i/n$. Linearity of expectation shows that the mean of $X_b$ is also $\mu$. The independence assumption implies the variance of $X_b$ is the sum of the variances of its terms. Each such term $X_i/n$ has variance $\sigma^2/n^2$ (because the variance of a constant times a random variable is the constant squared times the variance of the random variable). We have $n$ identically distributed such variables to sum, so each term has that same variance. As a result, we get $n \sigma^2/n^2 = \sigma^2/n$ for the variance of the sample mean.

Usually we do not know $\sigma^2$ and so we must estimate it from the data. Depending on the setting, there are various ways to do this. The two most common, general-purpose estimates of $\sigma^2$ are the sample variance $s^2 = \frac{1}{n}\sum_i(X_i-X_b)^2$ and a small multiple of it, $s_u^2 = \frac{n}{n-1}s^2$ (which is an unbiased estimator of $\sigma^2$). Using either one of these in place of $\sigma^2$ in the preceding paragraph and taking the square root gives the standard error in the form of $s/\sqrt{n}$ or $s_u/\sqrt{n}$.

Answer 3

+1 to both @JoelW. & @MichaelChernick. I want to add a detail to @JoelW.'s answer. He notes that "we almost never have a direct estimate of the SEM", which is essentially true, but it's worth explicitly recognizing a caveat to that statement. Specifically, when a study compares multiple groups / treatments (for example, placebo vs. standard drug vs. new drug), an ANOVA is typically used to see if they are all equal. The null hypothesis is that each group has been drawn from the same population, and thus, all three means are estimates of the population mean. That is, the null hypothesis in a standard ANOVA assumes that you do have a direct estimate of the SEM. Consider the equation for the variance of the sampling distribution of means: $$ \sigma^2_{\bar x}=\frac{\sigma^2_{pop}}{n_j}, $$ where $\sigma^2_{pop}$ is the population variance, and $n_j$ is the number of groups. Although we don't usually perform the calculations in this way, we could simply use standard formulas to plug in estimated values, and with minimal algebraic reshuffling, form the $F$ statistic like so: $$ F=\frac{n_j\times s^2_{\bar x}}{s^2_{\text{pooled within group}}} $$ In this case, we really would be using the standard formula (only applied over the group means), that is: $$ s^2_{\bar x}=\frac{\sum_{j=1}^{n_j}(\bar x_j-\bar x_.)^2}{n_j-1}, $$ with $x_.$ being the mean of the group means.

In that we typically believe the null hypothesis is not true, @JoelW.'s point is right, but I work through this point, because I think the clarity it affords is helpful for understanding these issues.