Distinguishing between $\epsilon$ and $e$ in interaction with residual maker matrix $M$

Question

I've hit a small snag in working out some of the implications of the residual maker matrix $M$.

Through previous posts I've been able to understand the difference between the use of $e$ and $\epsilon$, however there is still something I don't understand. We have the residual maker matrix, $M = I-X(X'X)^{-1}X'$, and we also know that $e=My$ since

$e = y - \hat y = y - Py = y - X(X'X)^{-1}X'y = (I-X(X'X)^{-1}X')y = My$

My concern, however, comes with when we evaluate this line further. We have two equivalencies for $y$. In the matrix form of our model we have $y=X\beta + \epsilon$. However, since we can also represent our residual as $e=y-X\hat \beta$ given that $\hat y = X\hat \beta$, we can also define $y=X\hat \beta + e$. If we substitute either $y$ into $e= My$, while also knowing that

$MX=0$ since $(I-X(X'X)^{-1}X')X = X - X(X'X)^{-1}X'X = X- XI = X -X = 0$,

we get that $e$ can be represented both as

$e = My = M(X\hat \beta + e) = Me$

and

$e= My =M(X\beta + \epsilon) = M\epsilon$.

If $e=Me=M\epsilon$, does that not necessitate that $e=\epsilon$, at least in this model? I've seen lecture notes write either one of these equivalencies but never both at the same time, so at this point I feel as if I'm running in circles. Any insight would be greatly appreciated!

Answer 1

The residual-maker matrix $M$ (also called the "hat matrix" and usually denoted as $\mathbf{h}$ or $\mathbf{H}$ in may sources) is a projection matrix that projects vectors in $\mathbb{R}^n$ onto the column-space of the design matrix. You are correct that it yields the following projection outputs:

$$M \epsilon = e \quad \quad \quad \quad \quad Me = e.$$

However, since the projection is not an injective operation, it does not follow from this that $e = \epsilon$. Different vectors can be projected onto the same vector by the projection. In this case we see that the error vector $\epsilon$ projects onto the residual vector $e$, and if we apply the projection to the residual vector $e$, nothing happens (since this vector is already in the column-space of the design matrix).

It is useful to remember that projections involve a loss of information --- i.e., they are not invertible. In general, when you look at projections operating in a vector space, there are an infinite number of vectors that will all be projected onto the same vector. The fact that two vectors are projected onto the same vector does not mean that they were the same vector to start with. In this case, the error vector and the residual vector are both projected onto the same vector, but they are still different vectors prior to the projection.