Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have two factors $a,b \in R^M$ which I wish to regress out of my variables $Y \in R^{M \times N}$. $b$ is a function of the variables $Y$ and is defined as:

$b = \frac{1}{N}Y1$

Regressing out the variables serially, the formula I have is:

$$\begin{eqnarray} Y_{\perp a} &=& Y - m(m^Tm)^{-1}m^TY \\ Y_{\perp a,b} &=& Y_{\perp a} - b_{\perp a}(b_{\perp a}^Tb_{\perp a})^{-1}b_{\perp a}^TY_{\perp a} \\ \text{is equivalent to:} \\ Y_{\perp a,b} &=& Y - [a,b]([a,b]^T[a,b])^{-1}[a,b]^TY \end{eqnarray}$$

Is there a formal way to prove that regressing out $[a,b]$ simultaneously to get $Y_{\perp a,b}$ is equivalent to regressing out $a$, then calculating $b_{\perp a}$ on the residual, $Y_{\perp a}$ and regressing that out to get the same $Y_{\perp a,b}$ .

I've tested this empirically in matlab and this appears to be true, but I can't come up with a formal proof.

share|improve this question
Your notation is quite strange. It seems that you have dependent variable $Y$ and two independent variables $a$ and $b$. You then do the following: 1. Regress $a$ on $Y$ and get the residual $e_1$. 2. Regress $b$ on $a$ and get the residual $e_b$. 3. Regress $e_b$ on $e_1$. You want to find out how to prove that this is identical to residual when you regress $a$ and $b$ on $Y$? Am I correct? – mpiktas Sep 24 '12 at 12:50
Yes, that's correct. – Swiss Army Man Sep 24 '12 at 13:35
Somebody will come up with an answer, but I suggest either writing down the equations yourself (with 2 variables it is not that hard) or looking up partitioned inverse and its implications for linear regressions. – mpiktas Sep 24 '12 at 13:49

Turns out there is actually a theorem which proves this:

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.