# Why is it that $M_{X_2} M_X = M_X$ for two different residual makers?

I can't seem to figure this out algebraically, or intuitively.

Same with a result like $$M_X M_X = M_X$$ which I know is the idempotent property for a residual maker.

If I think of $$M_X X = 0$$, this makes sense to me, because it's saying the vector of least-squares residuals of regressing X on itself is 0 - because X perfectly predicts itself.

But what's the interpretation when we have two residual makers? What exactly is the regression?

• I have edited the title pls. see if this is what you meant. Also not that you have to specify the relationship between $X_2$ and $X$. Is it for example the case that $X = [X_1 X_2]$ hence $X_2$ being part of $X$ or what? Commented Mar 16, 2020 at 7:55
• Sorry for the ambiguity. The title is correct, and yes, X_2 is a column vector of X Commented Mar 16, 2020 at 8:21

If you take some vector $$y$$ and regress on $$X$$ you get residuals $$\hat \epsilon = M_Xy$$ which is why $$M_X$$ is sometimes referred to as the "residual maker matrix". Let assume you decide to take these residuals and regress them on $$X_2$$ where $$X = [X_1 \ X_2]$$ then the identity that $$M_{X_2}M_X=M_X$$ for $$X = [X_1 \ X_2]$$ tells you that the residuals $$\hat u$$ from that regression will be $$\hat u=M_{X_2}\hat \epsilon =M_{X_2} M_Xy = M_Xy = \hat \epsilon$$ the same.

You could say that the first regression takes all the variation in $$y$$ that systematically varies with $$X$$ and puts the unsystematic variation in $$\hat \epsilon$$. Since $$X_2$$ is included in $$X$$ there is no variation left in these residuals that systematically varies with $$X_2$$.

Proof

To show that $$M_{X_2}M_X=M_X$$ for $$X = [X_1 \ X_2]$$.

It should makes sense to you that $$P_X X = X,$$ which can be read as for any column in $$X$$ the predicted values from a regression of the column unto $$X$$ is the column itself.

And by partioning this implies that $$P_X [X_1 \ X_2] = [X_1 \ X_2]$$ hence $$P_XX_2 = X_2,$$ stating simply that since it holds for any column of $$X$$ it must hold for some selection of the columns in $$X$$ the selection being $$X_2$$.

Using this it is easy to show that $$P_{X_2}P_X=P_{X_2}$$. Consider that

$$P_{X_2}P_X=X_2(X_2^\top X_2)^{-1}X_2^{\top}P_X$$

and notice that the final part $$X_2^{\top}P_X$$ is simply $$(P_XX_2)^\top$$ due to symmetry of $$P_X$$. But you know from above that $$P_XX_2=X_2$$ so $$(P_XX_2)^\top = X_2^\top$$. It follows then that

$$P_{X_2}P_X=X_2(X_2^\top X_2)^{-1}X_2^{\top}P_X = X_2(X_2^\top X_2)^{-1}X_2^{\top} =P_{X_2}$$ Finally consider that

$$M_{X_2}M_X = (I-P_{X_{2}})(I-P_X) = I -P_{X} - P_{X_2}+ P_{X_2}P_X = I -P_{X} = M_X$$ using from above that $$P_{X_2}P_X=P_{X_2}$$.

• Thank you so much for your answer. Can you also write an econometrics textbook? I've never seen someone give such a clear explanation of what is going on. Instead I have to cross-reference three textbooks to check the behind-the-scene lin alg proofs and intuitive meaning behind the concepts being taught. Commented Mar 16, 2020 at 10:05
• I can't get over how good your answer is ffs im gonna cry. Where have you been my whole life Commented Mar 16, 2020 at 10:12
• The beauty derives from the underlying mathematical theory of projections and vector spaces. So in general the point is to think about what the matrices $P_X$ and $M_X$ does to a vector. There is a rich geometric interpretation Gilbert Strang lectures on youtube are a good source to understand vectorspaces and Davidson and McKinnon an advanced econometrics book has a chapter on regression where they give a geometric interpretation using these projection matrices $P_X$ and $M_X$. I am happy you liked the answer. Commented Mar 16, 2020 at 10:27
• I'll be sure to check them out. It's been a while since I did linear algebra. Unfortunately, I never really did grasp the geometric intuition, but Greene's appendix has helped a lot with that. Commented Mar 16, 2020 at 10:33