2
$\begingroup$

I have some loose intuition with this but I don't understand it

Say we have a scatterplot of data in 2 dimensions. Then we can propose a mean model where, for all $x_i \in X$ the estimated $y_i = E[Y]$ or the average value of $y$ irrespective of $x$.

Then why are the estimated $y_i$ by the regression line equal to $E[Y|X]$? I feel like these two things are related but I don't understand whether 1) they are or 2) how so.

$\endgroup$
3
  • 1
    $\begingroup$ It might help to read the section 1.1.2 Predicting One Random Variable from Another from the course notes/book The Truth About Linear Regression. Link. $\endgroup$ Commented Dec 16, 2023 at 8:42
  • $\begingroup$ In your intuitive idea of a scatter plot, both regression lines go through the mean of the data $(\bar x, \bar y)$, but one is horizontal while the other takes account of the slope of the observations (to minimise the sum of squares of residuals) $\endgroup$
    – Henry
    Commented Dec 16, 2023 at 9:14
  • $\begingroup$ Who said they’re equal? I would view our predicted $\hat y_i$ as estimates of the expected values. $\endgroup$
    – Dave
    Commented Dec 20, 2023 at 12:15

2 Answers 2

1
$\begingroup$

In the linear regression, we try to find the best predictor given the values of $X$. There is no exact solution for a least squares problem since we are dealing with an overdetermined system.

In this scenario, the best predictor of $Y$ is $E(Y|X)$, which means that I look at the values of Y for a given X and I look at the conditional expectation $E(Y|X=x_i)$(where $x_i$ is the value taken by $X$).

If we were to just take $E(Y)$, then for any value of $X$, the best predictor is always $E(Y)$. This is not good enough, as we can use the information about Xs to update the best predictor which then becomes $E(Y|X)$.

$\endgroup$
1
$\begingroup$

In a Gaussian regression model, the conditional expectation is also the conditional mode (and therefore a natural predictor)

Given that we have the information from $\mathbf{x}$ available, our prediction is improved by taking this information into account. In fact the true regression value $\mu_i \equiv \mathbb{E}(Y_i|\mathbf{x}_i)$ is not the predicted value, since it is unknown. The predicted value is actually an estimate of this true regression value, which is:

$$\hat{y}_i = \hat{\mathbb{E}}(Y_i|\mathbf{x}_i) = \mathbf{x}_i \hat{\boldsymbol{\beta}}.$$

As to the intuition for why this predicted value is a good one, if we have a regression model with normally distributed error terms (i.e., a Gaussian regression model) then the conditional expected value is also the mode of the conditional distribution, so our estimate of the conditional expected value is also an estimate of the mode of the conditional distribution. The mode is of course the value that has the highest conditional probability density for the response value to be predicted, so it is a natural point-based predictor for this value.$^\dagger$ If we were to form an interval-based prediction we would naturally start with the mode and expand outward to form a HDR around this point.

It is worth noting a situation where the best predicted value would not be the estimated conditional expectation. This situation could occur in a regression model where the error distribution is skewed, such that the mode of the distribution does not correspond to the mean. In such a case, the best point-based prediction of the response variable would lie off the regression line, instead occurring on the line of values corresponding to the conditional mode (rather than the conditional expected value).$^\dagger$


$^\dagger$ While this gives a reasonable heuristic explanation, a deeper explanation would arguably have reference to the pivotal quantity formed to predict the response variable. In a Gaussian linear regression with $n$ data points and $k+1$ parameters (including the intercept term) this pivotal quantity is:

$$\frac{Y_i - \hat{\mathbb{E}}(Y_i|\mathbf{x}_i)}{\hat{\mathbb{S}}(Y_i|\mathbf{x}_i)} \sim \text{St}(\text{df} = n-k-1).$$

As with the standard normal distribution, the Student's-T distribution also has its mode at zero, so the mode of the pivotal quantity occurs when $\hat{y}_i - \hat{\mathbb{E}}(Y_i|\mathbf{x}_i) = 0$, which again gives $\hat{y}_i = \hat{\mathbb{E}}(Y_i|\mathbf{x}_i)$. Again, it is noteable that if the pivotal quantity were to have a skewed distribution where the mode does not correspond to the mean then the mode would be the best point-based predictor and so the prediction would not be equal to the conditional mean.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.