Why is the formula for standard error the way it is?

Question

So just "why" is $SE = \frac{s}{\sqrt n}$ ? How should one interpret/articulate the reason of having $\sqrt n$ in the denominator. Why do we divide sample mean by the square root of the sample size, intuitively speaking? And how/why is it called standard "error". (Question equally applicable for true standard deviation of the population: $\frac{\sigma}{\sqrt n}$)

Is there an intuitive derivation of $SE$ that can make this clear?

Please assume you are explaining it to a 6 year old who understands mean and sample size :)

Answer 1

This comes from the fact that $\newcommand{\Var}{\operatorname{Var}}\newcommand{\Cov}{\operatorname{Cov}}\Var(X+Y) = \Var(X) + \Var(Y) + 2\cdot\Cov(X,Y)$ and for a constant $a$, $\Var( a X ) = a^2 \Var(X)$.

Since we are assuming that the individual observations are independent the $\Cov(X,Y)$ term is $0$ and since we assume that the observations are identically distributed all the variances are $\sigma^2$. So

$\Var( \frac{1}{n} \sum X_i ) = \frac{1}{n^2} \sum \Var(X_i) = \frac{1}{n^2} \times \sum_{i=1}^n \sigma^2= \frac{n}{n^2} \sigma^2 = \frac{\sigma^2}{n}$

And when we take the square root of that (because it is harder to think on the variance scale) we get $\dfrac{\sigma}{\sqrt{n}}$.

More intuitively, think of 2 statistics classes: in the first the teacher assigns each of the students to draw a sample of size 10 from a set of tiles with numbers on them (the teacher knows the true mean of this population, but the students don't) and compute the mean of their sample. The second teacher assigns each of his/her students to take samples of size 100 from the same set of tiles and compute the mean. Would you expect every sample mean to exactly match the population mean? or to vary about it? Would you expect the spread of the sample means to be the same in both classes? or would the 2nd class tend to be closer to the population? That's why it makes sense to divide by a function of the sample size. The square root means we have a law of diminishing returns, to halve the standard error you need to quadruple the sample size.

As for the name, the full name is "The estimated standard deviation of the sampling distribution of x-bar"; it only takes saying that a few times before you appreciate having a shortened form. I don't know who first substituted "error" for "deviation" this way, but it stuck. The standard deviation measures variability of individual observations; the standard error measures variability in estimates of parameters (based on observations).

Answer 2

Why √n?

So there is this theory called the central limit theorem that tells us that as sample size increases, sampling distributions of means become normally distributed regardless of the parent distribution. In other words, given a sufficiently large sample size, the mean of all samples from a population will be the same as the population mean. We know that this really happens. Let’s put that in the bank and revisit it later.

Now, let's look at what happens when we have a small sample size (n=5), and then observe what happens if we increase the sample size and leave all the population parameters the same.

population mean: 15 standard deviation: 5 N=5

In the above case, our standard error of the mean (S.E.M.) will be: 5/√5 = 2.236

Now let's consider a case with a sample size of 10

population mean: 15 standard deviation: 5 N=10

S.E.M. will be: 5/√10 = 1.158

Increase the sample to 100: S.E.M = 5/√100 = .5

Increase the sample to 1000: S.E.M = 5/√1000 = .158

...You see the pattern. As sample size increases, the standard error of the mean decreases and will continue to approach zero as your sample size increases infinitely. Why is this? One way to think about it is that if you keep increasing your sample size, you will eventually sample the entire population, at which point, your sample mean is your population mean. That is, any given sample mean will probably not be exactly equal to the true population mean, but as your sample size increases toward the size of the entire population, the amount that a given sample mean is likely to be off by (the standard error) becomes smaller and smaller.

Now, let's go back to the conceptual definition of the standard error of the mean. One way to look at it is as the "standard deviation of sample means", or, alternatively, "On average, a sample of size N will deviate from the population mean by this amount". Therefore, your S.E.M. statistic is giving you an idea of how well your sample mean is likely to approximate the population mean.

So we know that the S.E.M. tells us how well a sample mean of size N approximates the population mean, and we also know that as our sample size increases, any given sample mean will more closely approximate the population mean.

The mathematical expression of those two ideas is the formula for S.E.M.

By dividing by the square root of N, you are paying a “penalty” for using a sample instead of the entire population (sampling allows us to make guesses, or inferences, about a population. The smaller the sample, the less confidence you might have in those inferences; that’s the origin of the “penalty”). That penalty is relatively large when your sample is very small. As the sample size increases, however, that penalty rapidly diminishes, infinitely approaching the point where your sample is equivalent to the population itself.

How fast does a sample mean approach equivalence with the population mean (i.e., how fast does the S.E.M. approach 0) as the sample size increases? That will depend on the numerator in the formula: standard deviation. The standard deviation is a measure of how predictable any given observation is in a population, or how far from the mean any one observation is likely to be. The less predictability, the higher the standard deviation. By the same token, the higher the standard deviation, the less quickly your sample mean will approach the population mean as sample size N increases. The power of the central limit theorem, however, shows us that as sample sizes become very large, the S.E.M. becomes very small, regardless of the standard deviation. That is, having a sufficiently large sample size will lead to a very small S.E.M. in just about all cases. The main differences in this trend between large or small standard deviations appears most notably when sample sizes are small – you pay a large penalty for having a small sample from a population that has a lot of variability!