# Unbiased least squares estimate for GM Theorem

In order to prove the Gauss-Markov Theorem, we first have to show that the OLS estimate $$\hat{\theta}$$ is an unbiased estimator. From what Im reading on Internet and some textbooks, these are the main steps (skipping some stages):

\begin{aligned} \mathbb{E}[\hat{\theta}] = \quad & \mathbb{E}[(X^TX)X^TY] \\ \stackrel{\ast}{=} \quad & \mathbb{E}[(X^TX)X^T(X\theta + \epsilon)] = \theta \end{aligned}

But I can't figure out why on the $$\stackrel{\ast}{=}$$ step we are allowed to replace $$Y$$ which from my understanding consists of a known and deterministic vector (actual known labels) with $$X\theta + \epsilon = \hat{Y}$$ (label prediction of known data points). $$Y=\hat{Y}$$ iff the linear model is assumed to be exact all the time. But the residual sum of squares goes very rarely (or never) to 0, consequently it should never be the case.

Could someone enlighten the situation? Because Im a bit confused right now. Thanks!

$$Y=X\theta+\varepsilon$$ is a linear model which splits off the zero-mean stochastic part $$\varepsilon$$ form the deterministic part $$X\theta$$ of $$Y$$. It is not generally true that $$\hat Y=X\theta+\varepsilon$$. Usually, we take $$\hat Y=X\theta$$, and this is an optimal point prediction under square loss.
• Im not sure to understand your answer as it doesn't seems to answer my original question. Taking $\hat{Y} = X\theta$ or $\hat{Y} = X\theta + \epsilon$ is not really the part that bothers me. In fact I really don't understand how, conceptually, we can replace $Y$ with $\hat{Y}$ in the $\stackrel{\ast}{=}$ step. For me, $Y$ comes from the OLS estimate closed formula (and in a real world problem is the provided data labels) whether $\hat{Y}$ is the model prediction itself (which is conceptually different from the ground truth).
• @akiro, we never replace $Y$ with $\hat Y$. $Y=X\theta+\varepsilon$ per definition of $\theta$ and $\varepsilon$, while $\hat Y=X\theta$. It is not $Y$ that comes from an OLS estimate, because $Y$ is the actual data. The only thing related to $Y$ that comes from OLS is the fitted values $\hat Y$. Commented Jan 18, 2022 at 11:25