Understanding Expected Value of observed responses in Regression

Question

I have a quick question about the following is a discussion on the online Penn State Regression methods course:

Recall that, in fitting a regression model to data, we attempt to estimate the average—or expected value—of the observed responses $E(y_i)$ at any given predictor value x. That is, $E(y_i)$ is the population regression function. Because the average of the observed responses depends on the value of x, we might also denote this population average or population regression function as $\mu_{Y|x}$

Now, if there is no bias in the predicted responses, then the average of the observed responses $E(y_i)$ and the average of the predicted responses $E(\hat y_i)$ both equal the thing we are trying to estimate, namely the average of the responses in the population $μ_{Y|x}$. On the other hand, if there is bias in the predicted responses, then $E(y_i)$ = $μ_{Y|x}$ and $E(\hat y_i)$ do not equal each other. The difference between $E(y_i)$ and $E(\hat y_i)$ is the bias $B_i$ in the predicted response, i.e

$B_i$=$E(y_i)−E(\hat y_i)$

(See "Bias in predicted responses" section here for context).

My question here is about the average/expected value of observed responses, $E(y_i)$ that we're talking about. Could someone intuitively explain to me what this means? How can observed y-values in a sample have a "true" population value, or am I looking at it the wrong way?

Answer 1

In the comment below, you write:

"We attempt to estimate the average/expected value of observed responses". What do they mean by this? As in, I'm trying to intuitively understand what the "Average" of observed responses would be?

This concept of "average" really clicked with me when I started modeling things explicitly in the Stan language (a statistical language for Bayesian inference). I'm going to be stealing some of my notation from their manual.

So, we can usually write a linear regression like this:

$y_i = \beta_0 + \beta_1x_i + \epsilon_i$ where $\epsilon_i \sim N(0, \sigma^2)$

A few things to note: $y$ is the observed dependent variable, $x$ is the observed independent variable, and $i$ represents that each individual has their own value for it. $\beta_0$ is the intercept and $\beta_1$ is the slope for $x$. $\epsilon$ is the residual term. We can see that we assume that the residuals are normally distributed with a mean of zero and some variance.

Now, this can be rewritten as:

$y_i - (\beta_0 + \beta_1x_i) \sim N(0, \sigma^2)$

Which we can further simplify to:

$y_i \sim N(\beta_0 + \beta_1x_i, \sigma^2)$

And lastly, since $\hat{y} = \beta_0 + \beta_1x_i$, we can simplify again:

$y_i \sim N(\hat{y_i}, \sigma^2)$

Notice the subscript $i$. This means that, for every individual, we say their observed score comes from a normal distribution with a mean of the predicted value and a given residual variance. Notice that, for every person, this predicted value is different, but the variance is the same. The variance being the same is where we get the assumption of homogeneity of variance.

Now, I don't really like using "average value" and "expected value" interchangeably, because for some distributions, they are not the same thing. But for the normal distribution, the average and expected values are the same (because the mean is equal to the mode of a normal distribution). But that is what they mean by average. You are basically saying:

"Every observation of y is distributed with a mean of y-hat and a variance of sigma squared." This only clicked for me when I started using Stan, because in that language, you can literally code it as:

y ~ normal(beta[0] + beta[1] * x, sigma)