2
$\begingroup$

I can't seem to figure this out algebraically, or intuitively.

Same with a result like $M_X M_X = M_X$ which I know is the idempotent property for a residual maker.

If I think of $M_X X = 0$, this makes sense to me, because it's saying the vector of least-squares residuals of regressing X on itself is 0 - because X perfectly predicts itself.

But what's the interpretation when we have two residual makers? What exactly is the regression?

$\endgroup$
2
  • $\begingroup$ I have edited the title pls. see if this is what you meant. Also not that you have to specify the relationship between $X_2$ and $X$. Is it for example the case that $X = [X_1 X_2]$ hence $X_2$ being part of $X$ or what? $\endgroup$ Commented Mar 16, 2020 at 7:55
  • $\begingroup$ Sorry for the ambiguity. The title is correct, and yes, X_2 is a column vector of X $\endgroup$ Commented Mar 16, 2020 at 8:21

1 Answer 1

2
$\begingroup$

If you take some vector $y$ and regress on $X$ you get residuals $\hat \epsilon = M_Xy$ which is why $M_X$ is sometimes referred to as the "residual maker matrix". Let assume you decide to take these residuals and regress them on $X_2$ where $X = [X_1 \ X_2]$ then the identity that $M_{X_2}M_X=M_X$ for $X = [X_1 \ X_2]$ tells you that the residuals $\hat u$ from that regression will be $\hat u=M_{X_2}\hat \epsilon =M_{X_2} M_Xy = M_Xy = \hat \epsilon$ the same.

You could say that the first regression takes all the variation in $y$ that systematically varies with $X$ and puts the unsystematic variation in $\hat \epsilon$. Since $X_2$ is included in $X$ there is no variation left in these residuals that systematically varies with $X_2$.

Proof

To show that $M_{X_2}M_X=M_X$ for $X = [X_1 \ X_2]$.

It should makes sense to you that $$P_X X = X,$$ which can be read as for any column in $X$ the predicted values from a regression of the column unto $X$ is the column itself.

And by partioning this implies that $P_X [X_1 \ X_2] = [X_1 \ X_2]$ hence $$P_XX_2 = X_2,$$ stating simply that since it holds for any column of $X$ it must hold for some selection of the columns in $X$ the selection being $X_2$.

Using this it is easy to show that $P_{X_2}P_X=P_{X_2}$. Consider that

$$P_{X_2}P_X=X_2(X_2^\top X_2)^{-1}X_2^{\top}P_X$$

and notice that the final part $X_2^{\top}P_X$ is simply $(P_XX_2)^\top$ due to symmetry of $P_X$. But you know from above that $P_XX_2=X_2$ so $(P_XX_2)^\top = X_2^\top$. It follows then that

$$P_{X_2}P_X=X_2(X_2^\top X_2)^{-1}X_2^{\top}P_X = X_2(X_2^\top X_2)^{-1}X_2^{\top} =P_{X_2}$$ Finally consider that

$$M_{X_2}M_X = (I-P_{X_{2}})(I-P_X) = I -P_{X} - P_{X_2}+ P_{X_2}P_X = I -P_{X} = M_X$$ using from above that $P_{X_2}P_X=P_{X_2}$.

$\endgroup$
4
  • $\begingroup$ Thank you so much for your answer. Can you also write an econometrics textbook? I've never seen someone give such a clear explanation of what is going on. Instead I have to cross-reference three textbooks to check the behind-the-scene lin alg proofs and intuitive meaning behind the concepts being taught. $\endgroup$ Commented Mar 16, 2020 at 10:05
  • $\begingroup$ I can't get over how good your answer is ffs im gonna cry. Where have you been my whole life $\endgroup$ Commented Mar 16, 2020 at 10:12
  • $\begingroup$ The beauty derives from the underlying mathematical theory of projections and vector spaces. So in general the point is to think about what the matrices $P_X$ and $M_X$ does to a vector. There is a rich geometric interpretation Gilbert Strang lectures on youtube are a good source to understand vectorspaces and Davidson and McKinnon an advanced econometrics book has a chapter on regression where they give a geometric interpretation using these projection matrices $P_X$ and $M_X$. I am happy you liked the answer. $\endgroup$ Commented Mar 16, 2020 at 10:27
  • $\begingroup$ I'll be sure to check them out. It's been a while since I did linear algebra. Unfortunately, I never really did grasp the geometric intuition, but Greene's appendix has helped a lot with that. $\endgroup$ Commented Mar 16, 2020 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.