5
$\begingroup$

From the Wikipedia definition - "a confounder (also confounding variable, confounding factor, or lurking variable) is a variable that influences both the dependent variable and independent variable, causing a spurious association."

So to my understanding, a confounder would usually be correlated with the main independent variable, as it influences it.

One way to control for a confounder would be to add it to the multiple regression model. But in the context of machine learning it is said that having correlated features in the model should be avoided. In particular, it was answered in the following question: https://datascience.stackexchange.com/questions/36404/when-to-remove-correlated-variables

"But if are concerned about interpretability then it might make sense to remove one of the variables, even if the correlation is mild. This is particularly true for linear models. One of the assumptions of the linear regression is lack of perfect multicollinearity in the predictors. If A is correlated with B, then you cannot interpret the coefficients of neither A nor B. To see why, imagine the extreme case when A=B (perfect correlation). Then, the model y=100A+50B is the same as the model y=5A+10B or y=-2000A+4000B. There are multiple equilibra in the possible solutions to the least square minimzation problem therefore you cannot "trust" neither."

So to my understanding, if the confounder we add to the multiple regression model is correlated(which to my understanding is usually the case) with the independent variable, we will not be able to interpret the coefficients appropriately, so how could we actually understand the relationship between the main independent variable and the dependent variable?

$\endgroup$
3
  • $\begingroup$ Another way to control for a confounding variable is to do your linear regressions separately for different values of the confounder. I'm not so sure you can't interpret the coefficients appropriately. $\endgroup$ Commented Sep 28, 2020 at 21:50
  • $\begingroup$ There's a difference between being correlated and being perfectly correlated. $\endgroup$ Commented Sep 28, 2020 at 21:53
  • $\begingroup$ Could you please explain how $100A + 50B = 5A + 10B = 2000A + 4000B?$ $\endgroup$
    – Dave
    Commented Sep 28, 2020 at 22:33

1 Answer 1

3
$\begingroup$

The statements

But if are concerned about interpretability then it might make sense to remove one of the variables, even if the correlation is mild.

and

One of the assumptions of the linear regression is lack of perfect multicollinearity in the predictors.

are unrelated to each other. Perfect multicolinearity is not the same as a correlation between predictors. You do not ever have to think about perfect multicollinearity because all regression software will drop one of a pair of perfectly colinear predictors (though if you're doing regression by hand, perfect multicolinearity will cause a problem in inverting the matrices). Perfect multicolinearity is a problem of making a mistake in specifying your model, not a statistical problem.

Correlated predictors are not a problem at all for regression because regression extracts the unique contribution of each predictor to the outcome. Because most of the loss functions used to estimate the coefficients have a global minimum and are convex, there is only one solution to the regression estimation problem and the issue of multiple coefficient vectors being compatible with the same solution is absent.

It is indeed true that the more correlated predictors are, there less precision there is in estimating their effect, precisely because there is less information available to distinguish the unique effects of the predictors from each other. But this is not something the analyst gets to control except in an experiment; correlations among observed predictors are a product of nature. Failing to include a predictor necessary to control confounding in a model because it is correlated with the focal predictor will leave your effect estimate biased (although there is some work on managing the bias-variance trade-off by dropping predictors with certain qualities; e.g., Wu et al. (2011)).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.