What does standard error of the mean ACTUALLY show?

Question

I'm brushing up on my stats, so please bare with me (and correct me) for any mistakes. I really hope someone can help me out!

Let's consider two separate experiments that are designed to measure the length of a string.

Experiment One - (Or: How I view a statistician would determine the length of a string)

Imagine I have a population of 500 strings. I randomly sample 20 strings from this population, measure their lengths, and calculate the sample mean. I then repeat this process 100 times. By the end of the experiment, I will have 100 means, one for each time I sampled the population. This is the sampling distribution of the mean.

As I understand it, the standard deviation of this sampling distribution is the standard error of the mean. We want the standard error of the mean to be small as it means we are better zeroed in on the true population mean.

However, the standard error of the mean is also expressed as the ratio of the standard deviation of the population to the square root of the sample size (here, 20). Furthermore, it can be estimated as the ratio of the standard deviation of a single sampling of 20 strings to the square root of that sample size (again, 20).

Question one:

So my question is, how does the second definition using only the standard deviations of the population or sample along with the sample size connect to the original definition in which standard error of the mean is defined as the standard deviation of our sampling distribution? I can't wrap my head around the connection.

For instance, as we conduct more and more samplings, the standard deviation of the resulting sampling distribution will continue to decrease more and more, right? So how is this fact taken into account in the equation that only uses standard deviation of a single sample divided by that sample size? Surely the standard deviation of the sampling distribution (which is the standard error!) consisting of 20000000 means will be smaller than the value we get if we simply calculate it by taking the ratio of a single sample standard deviation to the sqrt of the sample size, right?

Question two:

Using the second definition, we are calculating standard error by looking at a single sample consisting of 20 measurements. But this isn't even a sampling distribution of the mean, but rather a point estimate of the mean. So how is it possible for it to even have a standard error when it's just ONE estimate?

Experiment Two - (Or: How I view a chemist/physicist would measure the length of a string)

Suppose I have a single string. I then measure that string 20 times. That's it.

Question three: In this experiment, there isn't really a 'population' from which I'm sampling. I'm just measuring the same string over and over. So how am I supposed to calculate a standard error from this? If each sampling has sample is size one, then its impossible to calculate any means nor any sampling distribution of those means. Alternatively, if we assume the 20 measurements belonged to a SINGLE sampling, then I'm still not able to construct a sampling distribution of the means, since I only got ONE mean. Sure, I could calculate the standard error of the 20 measurements, but that's not standard error, it's just the standard deviation!!

Or is it? What is it??WHAT IS ANYTHING????

Answer 1

I agree that the nomenclature and the formulas for the standard mean can be confusing. (Not complicated, actually, just confusing.) Our highly-voted threads in the "standard-error" tag may be enlightening.

I'll take your question step by step.

Imagine I have a population of 500 strings. I randomly sample 20 strings from this population, measure their lengths, and calculate the sample mean. I then repeat this process 100 times. By the end of the experiment, I will have 100 means, one for each time I sampled the population. This is the sampling distribution of the mean.

Correct!

As I understand it, the standard deviation of this sampling distribution is the standard error of the mean. We want the standard error of the mean to be small as it means we are better zeroed in on the true population mean.

Also correct! This is actually the definition of the standard error of the mean (or actually of any estimated parameter!): any parameter estimate will have a distribution, and the standard deviation of this distribution is defined to be the standard error of the parameter.

However, the standard error of the mean is also expressed as the ratio of the standard deviation of the population to the square root of the sample size (here, 20). Furthermore, it can be estimated as the ratio of the standard deviation of a single sampling of 20 strings to the square root of that sample size (again, 20).

Almost. The first statement is not an alternative definition of the SE. It is a mathematical equality that holds under certain assumptions (which are in practice usually fulfilled) that the SE of the mean is equal to $\frac{\sigma}{\sqrt{n}}$. And the second statement is correct: you can estimate the SEM by using an estimate $\hat{\sigma}$ of $\sigma$.

So my question is, how does the second definition using only the standard deviations of the population or sample along with the sample size connect to the original definition in which standard error of the mean is defined as the standard deviation of our sampling distribution? I can't wrap my head around the connection.

As above: that the two are equal is not a question of competing definitions. It's a question of having one definition (as above) and a mathematical theorem than the SEM so defined is equal to $\frac{\sigma}{\sqrt{n}}$.

For instance, as we conduct more and more samplings, the standard deviation of the resulting sampling distribution will continue to decrease more and more, right? So how is this fact taken into account in the equation that only uses standard deviation of a single sample divided by that sample size? Surely the standard deviation of the sampling distribution (which is the standard error!) consisting of 20000000 means will be smaller than the value we get if we simply calculate it by taking the ratio of a single sample standard deviation to the sqrt of the sample size, right?

No. The sampling distribution of the mean depends on the distribution of the original data and on how many observations each separate mean is calculated from (i.e., $n$). It does not depend on how often you sample n points and calculate a mean. This is just drawing more and more samples from the sampling distribution of the mean. The SD of these samples will not decrease just because you draw more and more often.

Simulations in R are a great tool to understand stuff like this. For instance, you could draw 100, 1000, 10000, ... means, each based on $n$ observations from the original data, and you could observe that the SD of the means does not budge a lot. For instance, here are the standard deviations of 10, 50, 100, 500, 1000, 5000, 10000 means, each one based on $n=20$ observations of the original population. It's a flat line, up to variability (meta: we could also investigate the standard error of the estimate of the standard error of the means, but I don't think we want to go there right now...):

R code:

set.seed(1) # for reproducibility
string_lengths <- runif(500)
nn <- 20
n_means <- c(10,50,100,500,1000,5000,10000)
sds <- sapply(n_means,function(kk)sd(replicate(kk,mean(sample(string_lengths,nn,replace=TRUE)))))
plot(n_means,sds,type="o")

Using the second definition, we are calculating standard error by looking at a single sample consisting of 20 measurements. But this isn't even a sampling distribution of the mean, but rather a point estimate of the mean. So how is it possible for it to even have a standard error when it's just ONE estimate?

Per above: the standard error is not a property of an observation, but of a distribution. And we can happily estimate it from a single observation of the distribution of the means... because this single observation is in turn based on $n$ observations from the underlying distribution of the original data!

Suppose I have a single string. I then measure that string 20 times. That's it.

Question three: In this experiment, there isn't really a 'population' from which I'm sampling. I'm just measuring the same string over and over. So how am I supposed to calculate a standard error from this? If each sampling has sample is size one, then its impossible to calculate any means nor any sampling distribution of those means. Alternatively, if we assume the 20 measurements belonged to a SINGLE sampling, then I'm still not able to construct a sampling distribution of the means, since I only got ONE mean. Sure, I could calculate the standard error of the 20 measurements, but that's not standard error, it's just the standard deviation!!

Well, if you just wrote down a single observation 20 times, then you can't estimate the population standard deviation $\sigma$, because you have only one observation. (Technically, you have 20 observations, but they are not independent, which is one of the technical conditions I mentioned above. If your conditions are not met, of course all bets are off.) So in this situation, there is really nothing you can't do, and theory won't help you.

(Incidentally, there is a population you are sampling from. It may be the 500 strings we started out with, or it could be just a single one, but we always have a population. We are just not sampling from it independently.)