# Can you control for confounding variables by building a seperate primary and residual model?

TLDR: I want to seperate out any possible influence from a set of confounding variables on the response before estimating the effects of the variables of interest. Can I use two sequential models?

I am seeking to model a a probability of failure using approximately 2,000 features. I am using gradient boosted decision trees for this purpose. The primary goal of my modelling is to understand the effects of the most important features on the response. I am using plots of shap values to interpret these relationships. For example, I see that some variables are linearly associated with the response, while for others there is a step change at a particular value or over a certain range.

I am running into an issue with a particular set of confounding variables. In addition to the 2,000 variables of interest ($$X$$), there is a set of 20 variables ($$W$$) which I do not care about. I know that $$W$$ is highly related to the response, but since these variables are outside of our influence, they are not of primary interest. However, many features of $$X$$ are correlated with $$W$$, and even when $$W$$ are included in the model, a portion of their effect appears to be attributed to the features in $$X$$. I understand why this happens and that it is often the desirable outcome when two effects cannot be easily seperated. But, in my case, if it is possible an effect is driven by $$W$$, we want to attribute it to $$W$$ as fully as possible by default.

My inclination is to address this problem by creating two models. The first model would regress (using gradient boosted decision trees) the response onto $$W$$. Then, the residuals of this model, which would be unrelated to $$W$$, could be regressed onto $$X$$. In this way, it should be impossible for the effects of $$W$$ to be attributed to $$X$$.

My questions:

1. Is this a valid method for seperating out the effects of a set of confounding variables?
2. How might this method otherwise effect parameter estimates?
• You need to find out which way the causal information goes. If the variables in $X$ are causing the variables in $X_{\text{confound}},$ then you have a mediation situation, and not confounding. But if variables in $X_{\text{confound}}$ are causally influencing variables in $X,$ then you should control for them. You can control in various ways: stratification (analyze per values), back-door adjustment, instrumental variables, or front-door adjustment, depending on the exact situation. Commented Jan 20, 2021 at 23:16
• Stratification is generally applicable, for sure. Back-door and front-door are formulas having to do with probabilities, so they're limited in that way. IVs I'm unsure of; they are generally set up to work with linear regression, but obviously linear regression is only one ML algorithm. You could probably get some insight by running an ML algo from the IV to X, and then another ML algo from X to Y. But I'm not entirely sure of that. Commented Jan 21, 2021 at 17:13
• @AdrianKeister Thank you for the additional detail! It seems like the most applicable strategy would be stratification. In my case, unfortunately, it would be at least cumbersome, as there are around 20 possible confounding variables, not all of which are categorical. Commented Jan 22, 2021 at 20:28
• The two step method won't correctly account for the degrees of freedom. If you want to assign ambiguous variability in Y to X1 over X2, you can use a type I (sequential) sums of squares testing strategy. If you want to adjust for X1's effect on X2 in the estimate of X2's coefficient, SEM seems reasonable, as @AdrianKeister suggests. Commented Jan 22, 2021 at 20:44
• There is a certain amount of information in your dataset, which you can think of as indexed by the size of your dataset. You use 20 variables' worth of information in step 1, but the size of your dataset remains the same. So in step 2, it looks like you still have the full / original amount of information. This will bias estimates of variability (not necessarily of the location of the center) and make any subsequent inferences invalid. Commented Jan 25, 2021 at 14:12