Confused about the basics - distributions and standard error

Question

I've spent some time trying to understand what it is that statisticians are doing when they get a point estimate, and relate it back to some population parameter. I can do the calculations and interpret them, but I still don't really understand what underlies it.

Here is what I know, and where I'm stuck.

You are looking for some mean of something. In the true population, there will be a mean (µ) and a standard deviation from that mean (σ).

We obviously can't look at the entire population, but we take a sample of an amount of people (n) and for them, we find a mean (xbar) and standard deviation for them too (s).

Now, we want to try and estimate µ from our sample. xbar is our best estimate (the point estimate), but this will have some error. This is where I'm confused, this is why

• Our best estimate of the error is the standard error (the standard deviation of the sampling distributions), but I'm not sure why. Why do we use a sampling distribution at all when, in practice, we never take more than one sample?

Answer 1

The key point is that the sample mean has its OWN distribution (called the sampling distribution). You're right, we are not using to thinking along these lines because we rarly think about a collection of samples (we tend to just put all the data together to form one big sample).

How do we know this and what does it tell us? Well, there is a very deep theoretical result called the "Central Limit Theorem", that says when the sample size is large enough, the quantity

(xbar - $\mu$)/(standard error) $\sim$ N(0,1)

which tells you that xbar is very well approximated by a random variable having mean $\mu$ and a standard deviation which is its standard error.

Understanding the standard error now is the key to answering your question of why this result is important, because it answers a very fundamental question: why do we think the sample mean becomes a better estimate for the population mean as the sample size increases? The key point is that the standard deviation of xbar (the standard error) decreases with the sample size:

standard error = (standard deviation of x)/sqrt(n)

which tells us that the standard deviation of xbar decreases as the sample size increases, and so the range of values xbar can have become more and more concentrated around the population mean.