Statistics question: Why is the standard error, which is calculated from 1 sample, a good approximation for the spread of many hypothetical means? I'm re-learning statistics and got confused by the idea of taking the standard error from 1 sample's worth of data (standard deviation of sample divided by the square root of sample size, i.e. $\frac{s}{\sqrt{n}}$). As I understand it, the standard error is the spread of many sample means in an attempt to gauge how precise (not accurate) our estimate of the population mean is, but what if there's just the one sample? Usually experiments can't or just aren't repeated and only have 1 sample from a population, no? So, how can I pretend to know the spread of many means from samples of a population that I only did hypothetically?
If this is an example of a frequentist's versus a Bayesian approach to statistics, I'd just like to learn the frequentists side of the argument (just to understand what it is), please. Thanks in advance!
 A: 
As I understand it, the standard error is the spread of many sample means in an attempt to gauge how precise (not accurate) our estimate of the population mean is, but what if there's just the one sample?

Very short
A sample is not just one sample but contains many individual observations. Each of the observations can be considered as a sample (is there a difference between '$n$ samples of size 1' and '1 sample of size $n$'?). So you actually have multiple samples that can help to estimate the standard error in sample means.
In order to estimate the variance of the mean of samples, would you rather have a sample of size one million or multiple (say a hundred) samples of ten?
A bit longer
A sample will almost never be picked such that it perfectly matches the population. Sometimes a sample might pick relatively low values, sometimes a sample might pick relatively high values.
The variation in the sample mean, due to these random variations in picking the sample, is related to the variation in the population that is sampled. If the population has a wide spread in high and low values, than the deviations in a random samples with relatively high/low values will be corresponding to this wide spread and they will be large.
The error/variation in the means of samples relates to the variance of the population. So we can estimate the former with the help of an estimate of the latter. We can estimate the variance of sample.means by the variance of the population. And for this estimate of the variance of the population, one single sample is sufficient.
In formula form
The variance of the sample means $\sigma_n$ where the samples are of size $n$ is related to the variance of the population $\sigma$ $$\sigma_n = \frac{\sigma}{\sqrt{n}}$$
So an estimate of $\sigma$, for which a single sample is sufficient, can also be used to estimate $\sigma_n$.
A: Frequentist methods have a concept called a confidence procedure.  A confidence interval is an example of such a thing.  It is the procedure that we have confidence in, not so much the specific interval or point estimate.
If you were to perform an experiment very many times over the sample space, you would get many different estimates.  The estimators could be used to form a distribution of estimates.  That is called the sampling distribution of the estimator.  That distribution holds a predictable relationship with the population.
The standard error, as well as the sample mean, sample standard deviation, and so forth, are optimal procedures.  It begs the question of optimal at what?
Most of these estimators are the best unbiased estimator of the population parameter.  Usually, the definition of best is that it minimizes some form of loss function.  It answers the question of "what is the best estimator of the true value in the population, under the restriction that my estimator will be unbiased and minimize loss."
If you would repeatedly take samples from the population, then what you would find is that the sample estimate, for each sample, would approximately average to the population parameter.  However, without seeing any other samples, it is the best estimate.  If you had many samples then you could perform meta-analysis if you felt it necessary on the estimates collected up to that time.
The standard error is the best estimate from the data provided of the standard deviation of the sampling distribution of the estimator of interest.
A way to think about this is that every sample has signal and noise.  The goal of each procedure is to capture as much signal as is possible, subject to whatever constraints and rules you have in your optimization process, and to discard as much noise as is possible.  The sample standard error of the mean, or the sample mean, or the sample median, or the sample estimator of something are the best estimate of the population parameter for a given sample.
There is no Bayesian equivalent estimator because if a statistician uses a Bayesian procedure, then it just updates the posterior from the new sample without creating two estimates of the parameter of interest.
A sampling distribution is really an artifact of the procedure, such as attempting to find the population median.  It isn't so much a property of the population as a property of trying to find a parameter of the population.  It is a point estimate of how wide the sample estimates will be rather than how wide the population is.  Because the sampling distribution depends on how wide the population is there is a linkage between the descriptive statistics of the population and the descriptive statistics of the sampling process itself.
It is a bit dangerous to think in terms of precision.  Imagine a school with an even number of students and the school is large.  You flip a fair coin to put students in either group A or group B.  You notice that SE(A)>SE(B).
What aspect of tossing a coin made group B's estimate more precise?  Nothing of course.  It isn't more precise, it is just different.  Both are estimates of the sampling error, one just happens to be larger than the other.
The question of the usefulness of the standard error is another matter.  If you observe the left big toe of five randomly chosen people from around the world, then you will have less precision in your estimate than if you estimated from a sample size of 30 million people.  The standard error will tell you that you have about 2449 times less precision.  Is that useful?
