Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Hi, given this information we are meant to prove that the above estimator is unbiased. I understand the proof for the most part (below). What I do not understand is the intuitive reason why the projection matrix, or Mx, is in this proof. Can someone explain this to me? Also is there another way to show this estimator is unbiased?
Thanks!
Note 1: $M_X$ is not the projection matrix, but is called the residual maker for reasons I will make obvious below. Define $P_X = X(X'X)^{-1}X'$ as the projection matrix (into the column space of matrix $X$). Then, $M_X = (I-P_X)$. Also note that both $P_X$ and $M_X$ are symmetric and idempotent (i.e., $P_X'P_X = P_XP_X =P_X$).
Note 2: By definition, $\hat{\varepsilon} = Y - X\hat{\beta}$, where $\hat{\beta}$ denotes the OLS estimator of $\beta$. Now note that if we suppose that $Y = X\beta + \varepsilon$, then by how we have defined $M_X$, $M_XY = (Y - P_XY) = (Y - X\hat{\beta}) = \hat{\varepsilon}$. This is why we call $M_X$ residual maker in the first place.
Note 3: Now comes the interesting observation: By definition, ${\varepsilon} = Y - X{\beta}$. Thus, $M_X\varepsilon = M_X(X\beta - Y) = M_XX\beta - M_XY \overset{Note 2}{=} M_XX\beta - \hat{\varepsilon}$. Now all that is left to show is that $M_XX\beta = 0$, which is straightforward: $M_XX\beta = (I-P_X)X\beta = X\beta - P_XX\beta = X\beta - X(X'X)^{-1}X'X\beta = X\beta - X\beta = 0$.
Note 4: The intuitive reason that we need $M_X$ is that whilst ${\varepsilon}$ is unobservable, $\hat{\varepsilon}$ and $M_X$ are observables, so we can relate an unobservable quantity of interest to two observable quantities.
I hope this can help you out :)
