Population vs sample

When we think of linear regression, the implicit assumption is that we only observe a small fraction of a possibly infinite large population.

Thinking of simple averages, imagine a fair die. The expected value of this die is 3.5. This is the population average, because we actually observe the entire population in a sense. My question is the following: how and when can we distinguish between sample averages and population expected values?

• Well, more generally we assume a data-generating process as if the observations were sampled from an infinitely large population. With your die example, you might just say that symmetry & background knowledge make a certain distribution a plausible model for the data-generating process without reference to a population. But I don't understand your question: if you know what a sample average is, & what an expected value is, what more do you need to distinguish them? Commented Dec 15, 2015 at 15:44
• Let me rephrase my question: can we say that the sample average is an estimate of the expected value? Commented Dec 15, 2015 at 16:01
• If the distribution has an expectation (see Why does the Cauchy distribution have no mean?) then you can certainly say the sample mean is an estimator of it. But you can say that of any sample statistic: the question's how an good estimator it is & whether you can find a better one. So then you have to decide what defines the goodness of an estimator. An example of the sample mean's not being the best estimator of the expectation by any reasonable criterion is when $X$ has a uniform distribution between $0$ & unknown $\theta$. Commented Dec 15, 2015 at 16:46

You correctly computed the expected value of the random variable that gives the result of the fair die: 3.5. This tells you that if you roll the dice infinitely many times and take the mean of all the individual results, then you will get 3.5. An estimate of this quantity is given by the sample mean obtained by rolling the die a finite number of times and by computing the mean of the individual results.

As an illustration, let's roll the die 10 times

set.seed(1234)
n <- 10
x <- sample(x = 1:6, size = n, replace = TRUE, prob = rep(1, 6))
> x
[1] 2 5 5 5 1 5 2 3 5 5


and compute the sample mean:

mean(x)
[1] 3.8


Let's now roll the die many many times:

set.seed(1234)
n <- 10000
x <- sample(x = 1:6, size = n, replace = TRUE, prob = rep(1, 6))
> mean(x)
[1] 3.5171