# How to decorrelate one variable from another

the issue that (I think) I have is that I have 2 variables A and B that have some correlation. I want to use A for an analysis but ideally I would have a variable that is not correlated with B.

I know that I can use PCA two get two uncorrelated variables. However each of these two is very likely to have some correlation to the original B. How to Ingo about creating a variable X from A and B that is uncorrelated from B.

To avoid the X-Y problem I will provide some further information.

I want to do a fit/measurement using A (multivariate classifier) but in bin of B. But with A and B having a correlation this can bias my measurement.

• Welcome to Cross Validated! I'm failing to see how decorrelating the variable relates to your goal of evaluating a classifier. Could you please explain your rationale there?
– Dave
Commented Sep 20, 2023 at 15:50
• I am doing a particle physics analysis where I am doing a template fit of simulated histograms (of A) of different processes to measured data. The templates are fixed. We are doing two measurements (of X) in two regions depending on variable B. There is the expectation that there is a correlation between X and B. If there is also a dependence between A and B we might Introduce a bias on X. We checked with different fake dependencies between X and B and the bias is acceptable but ideally I would eliminate it.
– J.N.
Commented Sep 20, 2023 at 18:36

A lot of analysts are taught that "correlation of predictors" is a problem, but they're not told why. I can offer an interesting math fact. Because regression is a projection, there is no difference in terms of prediction between fitting a model:

$$E[Y|A,B] = \beta_0 + \beta_1 A + \beta_2 B$$

And

$$E[Y|A, B^*] = \gamma_0 + \gamma_1 A + \beta_2 B^*$$

Where $$B^* = (I - (A^TA)^{-1}A^T)B$$ (I assume centering WLOG similar to notation in Seber Lee). That is, replace $$B$$ with $$B^*$$, the residuals of regressing B on A, a vector that is orthogonal to $$A$$.

In a causal modeling framework, if $$B$$ is a mediator between $$A$$ and $$Y$$ then the $$\gamma_1$$ represents the total effect of $$A$$ whereas the $$\beta_1$$ represents the direct effect of $$A$$. If $$B$$ is a confounder, the second expression reintroduces the confounding bias and should not be used.

There are two possible takeaways from this math-fact:

1. Replacing variables with residuals is a powerful tool and allows analysts to fit complicated structural equation models with simple OLS. Or...

2. Correlation between predictors in and of itself doesn't matter. For prediction and inference, if B is a confounder you prefer the conditional effect $$\beta_1$$ to the marginal effect $$\gamma_1$$ due to confounding bias.