1
$\begingroup$

I'm learning applications on Central Limit Theorem and got really confused with a few points. Think of an example of applying Central Limit Theorem:

  1. We have a whole population of 10 billion items
  2. It's not possible to measure the whole population, so we take a sample from it instead. Our sample size is 10000, meaning that we randomly select 10000 items from the whole population. We can calculate the sample mean, which is the mean of these 10000 items
  3. We repeat step 2, say 8888 times, and we get 8888 samples, each has 10000 randomly selected items; We therefore also have 8888 sample mean values.

OK. Now there are 3 places where we can take standard deviations and I'm really confused with their relationship to each other:

value #1: the standard deviation of the whole population, 10 billion items.

value #2: the standard deviation within one sample, or the SD of 10000 randomly selected items.

value #3: the standard deviation of 8888 sample means.

I think when people talk about applying the Central Limit Theorem and the equation of "standard deviation" and "standard error":

SE = SD / sqrt(n)

, the SD refers to value #1 and SE refers to value #3, and n refers to sample size of 10000 in the above example.

So, is value #2 totally irrelevant in the story? Is it something we should never care about??

Improve this question
$\endgroup$
4
  • $\begingroup$ The standard deviation of the population is the true value, you either know it or not. Estimators are used to estimate it. The underlying distribution might be, or might not be Gaussian, e.g. for skewed distributions the mean, mode and median are different, so the sd might be quite shifted. The formula for SE is the SE of the mean; the SD is the SD of the distribution of your n=8888 means. $\endgroup$
    – corey979
    Commented Apr 26, 2019 at 6:34
  • $\begingroup$ @corey979 Thanks, but I'm confused: what's the difference between SE of the mean vs SD of those 8888 means? I've re-editted my question with more clear names. Could you point out which is value #1 and which is value #2 ? $\endgroup$
    – Kid_Learning_C
    Commented Apr 26, 2019 at 6:43
  • $\begingroup$ SD, in general, is a characteristic of a distribution, roughly speaking: how wide it is. SE of the mean says how precise the estimation of the mean is. Consider two exact Gaussians, $N(0,1)$ and $N(0,100)$. The means are known exactly in both cases, one cannot write it's $0\pm 1$ neither $0\pm 100$. Now consider a sample, say of two points. They can well be drawn from the tail of the distribution, so those could be $(40,60)$. The mean is 50, how close is it to the true value, zero? Consider also $(-10,100)$, or $(-10,10)$ etc. The more data you draw, the better the mean is constrained, (...) $\endgroup$
    – corey979
    Commented Apr 26, 2019 at 7:18
  • $\begingroup$ (...) i.e. the error, uncertainty about its exact value is smaller and smaller. In a quite flat distribution, like $N(0,100)$, it's relatively less ambiguous where the mean is when compared to $N(0,1)$. So SD is a feature of a distribution as a whole, and SE is the measure of uncertainty of a single parameter of it. $\endgroup$
    – corey979
    Commented Apr 26, 2019 at 7:20

2 Answers 2

Reset to default
1
$\begingroup$

You're correct in saying SD refers to SD #1 and SE refers to SD #3. SD #2 comes into play when SD #1 is not known. We can approximate SD #1 using SD #2. This is what motivates the t-test and t-distribution. When discussing the central limit theorem, though, SD #2 is indeed not relevant.

Improve this answer
$\endgroup$
5
  • $\begingroup$ You can also predict SD #3 from SD #2 by dividing by $\sqrt{1000}$ (again an approximation) $\endgroup$
    – Henry
    Commented Apr 26, 2019 at 8:09
  • $\begingroup$ @Henry What does the SD of a sample have to do with an SD of the means of (lots of) samples? $\endgroup$
    – corey979
    Commented Apr 26, 2019 at 9:41
  • $\begingroup$ @corey979. This R code illustrates my point set.seed(1); samplesize <- 1000; cases <- 8888; realmean <- 77; realsd <- 6; matdat <- matrix(rnorm(samplesize * cases, realmean, realsd), ncol=samplesize); firstsamplesd <- sd(matdat[1,]); samplemeans <- rowMeans(matdat) so the SD of the first sample firstsamplesd # 5.9395 is close to the realsd # 6 and thus you can predict the standard error of the mean as firstsamplesd / sqrt(samplesize) # 0.1878 and this is not far from sd(samplemeans) # 0.1893 or the desired $\frac{6}{\sqrt{1000}} \approx 0.1897$ $\endgroup$
    – Henry
    Commented Apr 26, 2019 at 11:28
  • $\begingroup$ @Noah thank you for your answer. An important follow-up questions is: How accurately can we approximate SD #1 using SD #2? Of course the bigger the sample size, the better the approximation. But it must require some strict mathematics to support this approximation. Is there any? Or do people just use SD#2 to approximate SD#1 assuming it's valid? $\endgroup$
    – Kid_Learning_C
    Commented Apr 26, 2019 at 15:43
  • $\begingroup$ There is no universal rule of thumb (I might say 1000?), but when you're using SD #2 to approximate SD #1, you us a t-distribution, which relies on degrees of freedom to adjust the shape of the distribution to account for the uncertainty of the estimation of SD #1. $\endgroup$
    – Noah
    Commented Apr 26, 2019 at 17:06
1
$\begingroup$

The first thing to note is that the mean and std-dev of a "population" is a universal constant, and does not change. When we are taking samples, we are actually trying to "estimate" these constants so called parameters.

Coming to the CLT interpretation, when you take a lot of samples, and then take the mean of those samples, you get a distribution of the means themselves, and std-dev of that is what value#3/SE refers. This gets reduced by the formula you provided as n (number of samples) get increased.

Now, the value#2 is just the std-dev of the "sample" not the mean, because you have just one/single mean for that single set of sample.

Let me know in the comments, if you still need clarification!

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.