0
$\begingroup$

In order to prove the Gauss-Markov Theorem, we first have to show that the OLS estimate $\hat{\theta}$ is an unbiased estimator. From what Im reading on Internet and some textbooks, these are the main steps (skipping some stages):

\begin{equation} \begin{aligned} \mathbb{E}[\hat{\theta}] = \quad & \mathbb{E}[(X^TX)X^TY] \\ \stackrel{\ast}{=} \quad & \mathbb{E}[(X^TX)X^T(X\theta + \epsilon)] = \theta \end{aligned} \end{equation}

But I can't figure out why on the $\stackrel{\ast}{=}$ step we are allowed to replace $Y$ which from my understanding consists of a known and deterministic vector (actual known labels) with $X\theta + \epsilon = \hat{Y}$ (label prediction of known data points). $Y=\hat{Y}$ iff the linear model is assumed to be exact all the time. But the residual sum of squares goes very rarely (or never) to 0, consequently it should never be the case.

Could someone enlighten the situation? Because Im a bit confused right now. Thanks!

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

$$ Y=X\theta+\varepsilon $$ is a linear model which splits off the zero-mean stochastic part $\varepsilon$ form the deterministic part $X\theta$ of $Y$. It is not generally true that $\hat Y=X\theta+\varepsilon$. Usually, we take $\hat Y=X\theta$, and this is an optimal point prediction under square loss.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Im not sure to understand your answer as it doesn't seems to answer my original question. Taking $\hat{Y} = X\theta$ or $\hat{Y} = X\theta + \epsilon$ is not really the part that bothers me. In fact I really don't understand how, conceptually, we can replace $Y$ with $\hat{Y}$ in the $\stackrel{\ast}{=}$ step. For me, $Y$ comes from the OLS estimate closed formula (and in a real world problem is the provided data labels) whether $\hat{Y}$ is the model prediction itself (which is conceptually different from the ground truth). $\endgroup$
    – Dime
    Commented Jan 18, 2022 at 9:28
  • $\begingroup$ @akiro, we never replace $Y$ with $\hat Y$. $Y=X\theta+\varepsilon$ per definition of $\theta$ and $\varepsilon$, while $\hat Y=X\theta$. It is not $Y$ that comes from an OLS estimate, because $Y$ is the actual data. The only thing related to $Y$ that comes from OLS is the fitted values $\hat Y$. $\endgroup$
    – Richard Hardy
    Commented Jan 18, 2022 at 11:25
  • $\begingroup$ Much clearer now, thank you! $\endgroup$
    – Dime
    Commented Jan 18, 2022 at 12:56
  • $\begingroup$ @akiro, you are welcome! $\endgroup$
    – Richard Hardy
    Commented Jan 18, 2022 at 14:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.