# Understanding Expected Value of observed responses in Regression

I have a quick question about the following is a discussion on the online Penn State Regression methods course:

Recall that, in fitting a regression model to data, we attempt to estimate the average—or expected value—of the observed responses $E(y_i)$ at any given predictor value x. That is, $E(y_i)$ is the population regression function. Because the average of the observed responses depends on the value of x, we might also denote this population average or population regression function as $\mu_{Y|x}$

Now, if there is no bias in the predicted responses, then the average of the observed responses $E(y_i)$ and the average of the predicted responses $E(\hat y_i)$ both equal the thing we are trying to estimate, namely the average of the responses in the population $μ_{Y|x}$. On the other hand, if there is bias in the predicted responses, then $E(y_i)$ = $μ_{Y|x}$ and $E(\hat y_i)$ do not equal each other. The difference between $E(y_i)$ and $E(\hat y_i)$ is the bias $B_i$ in the predicted response, i.e

$B_i$=$E(y_i)−E(\hat y_i)$

(See "Bias in predicted responses" section here for context).

My question here is about the average/expected value of observed responses, $E(y_i)$ that we're talking about. Could someone intuitively explain to me what this means? How can observed y-values in a sample have a "true" population value, or am I looking at it the wrong way?

• Do you mean residual instead of bias? Bias generally refers to a parameter being different from its true value; residuals refer to predicted values being different from their observed values. Jan 9, 2018 at 4:41
• Not bias. Sorry, I think I wrote the question a bit unclear. In the first line, it says "We attempt to estimate the average/expected value of observed responses". What do they mean by this? As in, I'm trying to intuitively understand what the "Average" of observed responses would be? Jan 9, 2018 at 4:46
• No apologies necessary! I was just clarifying to make sure that I understood the question correctly. Jan 9, 2018 at 4:47
• What part of the question did you write, and what part of the question was copied from the course? Jan 9, 2018 at 4:49

In the comment below, you write:

"We attempt to estimate the average/expected value of observed responses". What do they mean by this? As in, I'm trying to intuitively understand what the "Average" of observed responses would be?

This concept of "average" really clicked with me when I started modeling things explicitly in the Stan language (a statistical language for Bayesian inference). I'm going to be stealing some of my notation from their manual.

So, we can usually write a linear regression like this:

$y_i = \beta_0 + \beta_1x_i + \epsilon_i$ where $\epsilon_i \sim N(0, \sigma^2)$

A few things to note: $y$ is the observed dependent variable, $x$ is the observed independent variable, and $i$ represents that each individual has their own value for it. $\beta_0$ is the intercept and $\beta_1$ is the slope for $x$. $\epsilon$ is the residual term. We can see that we assume that the residuals are normally distributed with a mean of zero and some variance.

Now, this can be rewritten as:

$y_i - (\beta_0 + \beta_1x_i) \sim N(0, \sigma^2)$

Which we can further simplify to:

$y_i \sim N(\beta_0 + \beta_1x_i, \sigma^2)$

And lastly, since $\hat{y} = \beta_0 + \beta_1x_i$, we can simplify again:

$y_i \sim N(\hat{y_i}, \sigma^2)$

Notice the subscript $i$. This means that, for every individual, we say their observed score comes from a normal distribution with a mean of the predicted value and a given residual variance. Notice that, for every person, this predicted value is different, but the variance is the same. The variance being the same is where we get the assumption of homogeneity of variance.

Now, I don't really like using "average value" and "expected value" interchangeably, because for some distributions, they are not the same thing. But for the normal distribution, the average and expected values are the same (because the mean is equal to the mode of a normal distribution). But that is what they mean by average. You are basically saying:

"Every observation of y is distributed with a mean of y-hat and a variance of sigma squared." This only clicked for me when I started using Stan, because in that language, you can literally code it as:

y ~ normal(beta[0] + beta[1] * x, sigma)

• Thanks, that definitely did make sense! So then it would mean that the $E(y_i)$ = $\hat y_i$. Then what does the author mean when they write $E(\hat y_i)$ ? (Added bit of info, this explanation comes up when they're trying to explain the concept of mallow's cp.) Jan 9, 2018 at 5:24
• Well, I'm under the impression that: $E(y_i | x_i) = \hat{y}$. I don't know much about Mallow's Cp—besides what it is meant to do—and the Wikipedia page for it does act like the predicted value of $y$ and the value of $y$, given $x$, are different things. So I'm afraid I'm not much help at this point. Jan 9, 2018 at 6:30
• @ricksanchez Ah. I read through part of that chapter. $E(y)$ refers to what the actual, population regression line would predict. $E(\hat{y})$ refers to what our fitted, sample regression line predicts. As the author of that course notes, we don't know $E(y)$: "Did you notice all of those Greek parameters... [those] are generally used to denote unknown population quantities. That is, we don't or can't know about the value of $\Gamma_p$—we must estimate it." Since you don't know $E(y)$ (which is a part of $\Gamma_p$), Mallow's $C_p$ estimates it. Jan 9, 2018 at 6:51
• @ricksanchez So, in the words of that course: The residual value is how far our predicted value is from our observed value. And bias is how far our predicted value from the sample regression line is from the predicted value from the true, population regression line. We obviously do not know the true population regression line, so it is estimated by Mallow's $C_p$. I apologize—I got hung-up on terminology I wasn't familiar with; my original comment trying to clarify if you meant "residual" was incorrect—bias was the correct word. I just needed some context to figure it out. Jan 9, 2018 at 6:54