8
$\begingroup$

I'm re-learning statistics and got confused by the idea of taking the standard error from 1 sample's worth of data (standard deviation of sample divided by the square root of sample size, i.e. $\frac{s}{\sqrt{n}}$). As I understand it, the standard error is the spread of many sample means in an attempt to gauge how precise (not accurate) our estimate of the population mean is, but what if there's just the one sample? Usually experiments can't or just aren't repeated and only have 1 sample from a population, no? So, how can I pretend to know the spread of many means from samples of a population that I only did hypothetically?

If this is an example of a frequentist's versus a Bayesian approach to statistics, I'd just like to learn the frequentists side of the argument (just to understand what it is), please. Thanks in advance!

Improve this question
$\endgroup$
11
  • $\begingroup$ If you knew the original population had mean $\mu$ and variance $\sigma^2$ then you could say the mean of sample size $n$ would have expectation value $\mu$ and variance $\frac1n\sigma^2$ so standard error of the sample mean $\frac1{\sqrt n}\sigma$ (at least if you were sampling with replacement or from a distribution - the standard error will be smaller than this if sampling from a finite population without replacement). Your sample has a sample variance $s^2$ and you can use this as an estimator of $\sigma^2$ and thus calculate an estimate of the standard error of the sample mean. $\endgroup$
    – Henry
    Oct 24, 2021 at 2:03
  • $\begingroup$ But if your sample size is $1$, this does not work as you cannot calculate the sample variance or make a reasonable estimate of the population variance, at least not without other information $\endgroup$
    – Henry
    Oct 24, 2021 at 2:06
  • $\begingroup$ But what's the reasoning behind being able to use the sample variance s2 as an estimator of σ2? Because a sample of sufficient n (n is sufficiently large enough that it represents the population, i.e. captures much of the variation?) - that it's just as good as many hypothetical samples of a smaller sample size? $\endgroup$
    – Nate
    Oct 24, 2021 at 2:09
  • $\begingroup$ But you can and people do. All you need is the standard deviation of 1 sample and it's sample size, no? Btw just to be clear, I mean 1 sample of sufficient sample size (>30), not a sample size of 1. $\endgroup$
    – Nate
    Oct 24, 2021 at 2:13
  • $\begingroup$ Is the idea linked to the Central limit theorem, maybe? Where all distributions of sample means from any distribution of data is normally distributed? So, the SE calculated from 1 sample "shouldn't" be far off from the spread of many means? I'm obviously paraphrasing here, but just trying to get a general idea. $\endgroup$
    – Nate
    Oct 24, 2021 at 2:17

3 Answers 3

Reset to default
4
$\begingroup$

As I understand it, the standard error is the spread of many sample means in an attempt to gauge how precise (not accurate) our estimate of the population mean is, but what if there's just the one sample?

Very short

A sample is not just one sample but contains many individual observations. Each of the observations can be considered as a sample (is there a difference between '$n$ samples of size 1' and '1 sample of size $n$'?). So you actually have multiple samples that can help to estimate the standard error in sample means.

In order to estimate the variance of the mean of samples, would you rather have a sample of size one million or multiple (say a hundred) samples of ten?

A bit longer

A sample will almost never be picked such that it perfectly matches the population. Sometimes a sample might pick relatively low values, sometimes a sample might pick relatively high values.

The variation in the sample mean, due to these random variations in picking the sample, is related to the variation in the population that is sampled. If the population has a wide spread in high and low values, than the deviations in a random samples with relatively high/low values will be corresponding to this wide spread and they will be large.

The error/variation in the means of samples relates to the variance of the population. So we can estimate the former with the help of an estimate of the latter. We can estimate the variance of sample.means by the variance of the population. And for this estimate of the variance of the population, one single sample is sufficient.

In formula form

The variance of the sample means $\sigma_n$ where the samples are of size $n$ is related to the variance of the population $\sigma$ $$\sigma_n = \frac{\sigma}{\sqrt{n}}$$

So an estimate of $\sigma$, for which a single sample is sufficient, can also be used to estimate $\sigma_n$.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Aha! Ok, so if I may try to summarize this in my own words to see if I really get it...The argument being made is that if you have one sample that sufficiently captured the natural variability of a population with each individual collected, then a set of hypothetical re-samples with many means would be (slightly) redundant. It's a nice visual, but sometimes not possible to sample one thousand times. We already have an idea of what the spread of means would look like (maybe) because what's true of the sample is true of the population? $\endgroup$
    – Nate
    Oct 25, 2021 at 14:29
  • $\begingroup$ I think that captures it. I liked your question because it is indeed at first glance a bit counterintuitive. Typically you can estimate the variance by repeating several times, but in this case somehow you can estimate with a single repetition. (The reason is that with a single sample you do not really have a single repetition, instead you have many observations from which you can get information about the variance of the distribution and information about the variance of the estimate of the mean.) $\endgroup$
    – Sextus Empiricus
    Oct 25, 2021 at 15:01
  • $\begingroup$ Great explanation, thanks again! $\endgroup$
    – Nate
    Oct 25, 2021 at 15:11
3
$\begingroup$

I propose to put some visuals/intuition to your question... using an empirical approach (bootstrapping) to make it more concrete, especially in reference to the following:

Usually experiments can't or just aren't repeated and only have 1 sample from a population

As you highlighted it, we are talking about the standard error of a statistic (the mean in our case). So, Let's assume that you have a random sample of 20 people's height from a given country:

##  [1] 192.3214 144.4797 151.3796 155.2519 147.5844 147.9056 171.1867 159.3074
##  [9] 163.0097 190.9857 165.8155 198.2192 192.2418 165.3628 186.9498 167.3355
## [17] 148.6400 156.6933 160.8472 174.4827

From this sample, you get a mean of 167 and a standard deviation of 17.

You have only one random sample, but you can imagine that if you could take another one, you might get similar values, sometimes duplicates or sometimes more extreme values... but something that will look like to your initial random sample.

So, from these initial sample values and without inventing new ones (only resampling with replacement), you can imagine many other samples. For example, we can imagine three as follows:

##  [1] 165.8155 159.3074 148.6400 165.3628 155.2519 151.3796 192.2418 163.0097
##  [9] 159.3074 192.2418 186.9498 163.0097 144.4797 198.2192 159.3074 190.9857
## [17] 165.3628 159.3074 167.3355 156.6933

##  [1] 147.5844 147.9056 151.3796 163.0097 167.3355 159.3074 167.3355 156.6933
##  [9] 156.6933 159.3074 147.9056 190.9857 192.2418 171.1867 198.2192 147.9056
## [17] 155.2519 167.3355 148.6400 165.8155

##  [1] 192.2418 198.2192 156.6933 192.3214 148.6400 192.3214 198.2192 165.8155
##  [9] 167.3355 144.4797 163.0097 148.6400 159.3074 163.0097 163.0097 174.4827
## [17] 165.3628 165.8155 174.4827 159.3074

Their respective mean will be different from the initial one... but what is interesting is that if we repeat this resampling exercise 10,000 times, for instance, and we calculate the mean for each of these generated samples, we will get something like that (leaving the R code here, just to illustrate it), a distribution of means centered around the initial sample mean:

set.seed(007)

spl <- 167+17*scale(rnorm(20))[,1]  #Forcing to have same mean and sd for all samples

library(boot)
myFunc <- function(data, i){
  return(mean(data[i]))
}

bootMean <- boot(spl , statistic=myFunc, R=10000)
hist(bootMean$t, xlim=c(150,185), main="Sample size n=20")
abline(v=mean(spl), col="blue")

enter image description here

So, the histogram above represents the distribution of means of 10,000 samples… that we constructed from our initial sample. Empirically, we can determine the standard deviation of this (sampling) distribution (which is our standard error of the mean):

sd(bootMean$t)
## [1] 3.74095

Interestingly enough, if we calculate the formula for the standard error $\frac{s}{\sqrt n}$, we get something very similar:

sd(spl)/sqrt(20)
## [1] 3.801316

The standard error of the mean tells us about the spread our data around the mean.

To finish this intuitive overview, let's see what happen if we increase our initial sample size (to understand the impact of this $\sqrt{n}$).

enter image description here

enter image description here

So, if we increase the sample size, the standard error gets unsurprisingly smaller... we reduce the error in estimating the population mean. Again, we can empirically see that the formula still holds:

sd(bootMean$t)
## [1] 0.7740625
sd(spl)/sqrt(500)
## [1] 0.7602631
Improve this answer
$\endgroup$
3
  • $\begingroup$ I totally understand with bootstrapping, that makes sense - so thank you for the example! However, I always hear the phrase "IF we sampled many times...", as in we don't ever plan to. So, in my head I'm like, ok, so we still estimate with just one sample mean the spread of many sample means?? $\endgroup$
    – Nate
    Oct 25, 2021 at 15:21
  • $\begingroup$ @Nate, this "wording" comes from the Central Limit Theorem (en.wikipedia.org/wiki/Central_limit_theorem). "(...) If this procedure is performed many times, the central limit theorem says that the probability distribution of the average will closely approximate a normal distribution (...)". So, you do not need to actually sample multiple times since it is already encapsulated by the theorem. $\endgroup$
    – Pitouille
    Oct 25, 2021 at 15:52
  • $\begingroup$ Right, the shape of the sampling distribution of the mean makes sense as well. I got confused about how one sample mean could also tell you about the standard deviation or spread of many sample means. We might have collected a sample that is outside the bounds of a reasonable estimate of the mean (outside the 2 standard deviations region of a sampling distribution of the mean). Like one collection of houses somehow gives you a hint about the location on a map of all other houses - if you don't have the whole map, you could be totally wrong. $\endgroup$
    – Nate
    Oct 25, 2021 at 16:11
1
$\begingroup$

Frequentist methods have a concept called a confidence procedure. A confidence interval is an example of such a thing. It is the procedure that we have confidence in, not so much the specific interval or point estimate.

If you were to perform an experiment very many times over the sample space, you would get many different estimates. The estimators could be used to form a distribution of estimates. That is called the sampling distribution of the estimator. That distribution holds a predictable relationship with the population.

The standard error, as well as the sample mean, sample standard deviation, and so forth, are optimal procedures. It begs the question of optimal at what?

Most of these estimators are the best unbiased estimator of the population parameter. Usually, the definition of best is that it minimizes some form of loss function. It answers the question of "what is the best estimator of the true value in the population, under the restriction that my estimator will be unbiased and minimize loss."

If you would repeatedly take samples from the population, then what you would find is that the sample estimate, for each sample, would approximately average to the population parameter. However, without seeing any other samples, it is the best estimate. If you had many samples then you could perform meta-analysis if you felt it necessary on the estimates collected up to that time.

The standard error is the best estimate from the data provided of the standard deviation of the sampling distribution of the estimator of interest.

A way to think about this is that every sample has signal and noise. The goal of each procedure is to capture as much signal as is possible, subject to whatever constraints and rules you have in your optimization process, and to discard as much noise as is possible. The sample standard error of the mean, or the sample mean, or the sample median, or the sample estimator of something are the best estimate of the population parameter for a given sample.

There is no Bayesian equivalent estimator because if a statistician uses a Bayesian procedure, then it just updates the posterior from the new sample without creating two estimates of the parameter of interest.

A sampling distribution is really an artifact of the procedure, such as attempting to find the population median. It isn't so much a property of the population as a property of trying to find a parameter of the population. It is a point estimate of how wide the sample estimates will be rather than how wide the population is. Because the sampling distribution depends on how wide the population is there is a linkage between the descriptive statistics of the population and the descriptive statistics of the sampling process itself.

It is a bit dangerous to think in terms of precision. Imagine a school with an even number of students and the school is large. You flip a fair coin to put students in either group A or group B. You notice that SE(A)>SE(B).

What aspect of tossing a coin made group B's estimate more precise? Nothing of course. It isn't more precise, it is just different. Both are estimates of the sampling error, one just happens to be larger than the other.

The question of the usefulness of the standard error is another matter. If you observe the left big toe of five randomly chosen people from around the world, then you will have less precision in your estimate than if you estimated from a sample size of 30 million people. The standard error will tell you that you have about 2449 times less precision. Is that useful?

Improve this answer
$\endgroup$
5
  • $\begingroup$ I don't understand yet, maybe sleeping will help - haha. Thank you very much for the effort! $\endgroup$
    – Nate
    Oct 25, 2021 at 5:32
  • $\begingroup$ What are you looking for in the answer? $\endgroup$
    – Dave Harris
    Oct 25, 2021 at 5:38
  • $\begingroup$ This part was somewhat comforting: "The sample standard error of the mean... (is) ...the best estimate of the population parameter for a given sample." *It's the best we can do, given so little data. It's just odd, I thought of SE more like it was a range of "guesses" rather than a single number. There could be much more variability in a population and just using one sample doesn't seem to be a good estimate. *Range meaning it intuitively, to my mind, should require at least two samples. $\endgroup$
    – Nate
    Oct 25, 2021 at 5:56
  • $\begingroup$ You can know a range from a single sample, consider the German Tank Problem. en.wikipedia.org/wiki/German_tank_problem $\endgroup$
    – Dave Harris
    Oct 26, 2021 at 0:54
  • $\begingroup$ Oh cool, thank you! $\endgroup$
    – Nate
    Oct 26, 2021 at 1:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.