While there are questions regarding regressing out (or partialling out) a predictive variable, I want to regress out the dependent variable. I hope the question makes sense. Please let me know if this question was already asked, but I could not find it here.

(Im going to avoid using $X$ and $Y$ so that its not confusing). Suppose I have a matrix $X_1 \in R^{n\times d}$ and $X_2 \in R^n$ where $n$ denotes the number of samples and $d$ denotes the dimensionality of $X_1$. Suppose that $X_1$ has a linear relationship with $X_2$ and that $X_1$ is dependent on $X_2$. What I want to do is to "regress out" $X_2$ such that $X_1$ no longer contains information about $X_2$.

One approach I thought of is to simply run a linear regression for such that $X_1 = \beta X_2 + \epsilon$, and use $\epsilon$ for further inference, which lives in $R^{n\times d}$. I believe this is equivalent to running OLS on every column of $X_1$ individually with $X_2$ and then computing the residual.

Another approach I thought is to run a multiple regression, $X_2 = \beta X_1 + \epsilon$. We can think of $\beta: R^{n\times d} \rightarrow R^n$ as a map. Thus, we can compute the psuedo-inverse of $\beta$ to compute $\beta^{-1}X_2 - X_1 = \beta^{-1}\epsilon$. Again, $\beta^{-1}\epsilon$ lives in $R^{n\times d}$.

I guess the question is, are either of these methods valid? Or is there a better way to transform $X_1$?


Let $P = X_2(X_2'X_2)^{-1}X_2'$ be the $n \times n$ projection matrix for projecting onto $X_2$. The linear projection of the $d$ columns of $X_1$ onto $X_2$ is given by $PX_1$. The orthogonal component is: $$(I-P)X_1$$ This is your first approach.

| cite | improve this answer | |
  • $\begingroup$ This is just the closed form solution to the regression for first approach. Running some numerical experiments showed that in some cases, method 1 is equivalent to method 2, tho not always the case. $\endgroup$ – onepint16oz Nov 14 '18 at 0:47
  • $\begingroup$ @onepint16oz Your second method doesn't make sense to me. $\endgroup$ – Matthew Gunn Nov 14 '18 at 5:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.