# Regressing out the dependent variable

While there are questions regarding regressing out (or partialling out) a predictive variable, I want to regress out the dependent variable. I hope the question makes sense. Please let me know if this question was already asked, but I could not find it here.

(Im going to avoid using $$X$$ and $$Y$$ so that its not confusing). Suppose I have a matrix $$X_1 \in R^{n\times d}$$ and $$X_2 \in R^n$$ where $$n$$ denotes the number of samples and $$d$$ denotes the dimensionality of $$X_1$$. Suppose that $$X_1$$ has a linear relationship with $$X_2$$ and that $$X_1$$ is dependent on $$X_2$$. What I want to do is to "regress out" $$X_2$$ such that $$X_1$$ no longer contains information about $$X_2$$.

One approach I thought of is to simply run a linear regression for such that $$X_1 = \beta X_2 + \epsilon$$, and use $$\epsilon$$ for further inference, which lives in $$R^{n\times d}$$. I believe this is equivalent to running OLS on every column of $$X_1$$ individually with $$X_2$$ and then computing the residual.

Another approach I thought is to run a multiple regression, $$X_2 = \beta X_1 + \epsilon$$. We can think of $$\beta: R^{n\times d} \rightarrow R^n$$ as a map. Thus, we can compute the psuedo-inverse of $$\beta$$ to compute $$\beta^{-1}X_2 - X_1 = \beta^{-1}\epsilon$$. Again, $$\beta^{-1}\epsilon$$ lives in $$R^{n\times d}$$.

I guess the question is, are either of these methods valid? Or is there a better way to transform $$X_1$$?

Let $$P = X_2(X_2'X_2)^{-1}X_2'$$ be the $$n \times n$$ projection matrix for projecting onto $$X_2$$. The linear projection of the $$d$$ columns of $$X_1$$ onto $$X_2$$ is given by $$PX_1$$. The orthogonal component is: $$(I-P)X_1$$ This is your first approach.