How to decorrelate one variable from another

Question

the issue that (I think) I have is that I have 2 variables A and B that have some correlation. I want to use A for an analysis but ideally I would have a variable that is not correlated with B.

I know that I can use PCA two get two uncorrelated variables. However each of these two is very likely to have some correlation to the original B. How to Ingo about creating a variable X from A and B that is uncorrelated from B.

---

To avoid the X-Y problem I will provide some further information.

I want to do a fit/measurement using A (multivariate classifier) but in bin of B. But with A and B having a correlation this can bias my measurement.

Answer 1

A lot of analysts are taught that "correlation of predictors" is a problem, but they're not told why. I can offer an interesting math fact. Because regression is a projection, there is no difference in terms of prediction between fitting a model:

$$E[Y|A,B] = \beta_0 + \beta_1 A + \beta_2 B$$

And

$$E[Y|A, B^*] = \gamma_0 + \gamma_1 A + \beta_2 B^*$$

Where $B^* = (I - (A^TA)^{-1}A^T)B$ (I assume centering WLOG similar to notation in Seber Lee). That is, replace $B$ with $B^*$, the residuals of regressing B on A, a vector that is orthogonal to $A$.

In a causal modeling framework, if $B$ is a mediator between $A$ and $Y$ then the $\gamma_1$ represents the total effect of $A$ whereas the $\beta_1$ represents the direct effect of $A$. If $B$ is a confounder, the second expression reintroduces the confounding bias and should not be used.

There are two possible takeaways from this math-fact:

1. Replacing variables with residuals is a powerful tool and allows analysts to fit complicated structural equation models with simple OLS. Or...

2. Correlation between predictors in and of itself doesn't matter. For prediction and inference, if B is a confounder you prefer the conditional effect $\beta_1$ to the marginal effect $\gamma_1$ due to confounding bias.