3
$\begingroup$

Say you regress $Y = x_1,….x_k$ and find out that you have multicollinearity. I propose the following solution:

Say you suspect $x_1$ and $x_2$ are collinear. I regress $x_1$ on $x_2$ and get the residuals - call them $e_1$. Now instead of $x_1,x_2,..x_k$ in my original regression model, I use $x_1,e_1,…x_k$ in my original regression model. Will this approach work? If not, is there a better, related way?

$\endgroup$
4
  • $\begingroup$ The $e_1$ term isn't clear to me. Are you taking the residuals and entering them as a predictor in the regression? $\endgroup$ Commented Nov 5, 2023 at 9:59
  • $\begingroup$ It seems to me that that the parameter estimates for $e_1$ will be hard to interpret. There are already a few ways to deal with collinearity (e.g. ridge regression, various ways to combine the variables). Why don't you like them? $\endgroup$
    – Peter Flom
    Commented Nov 5, 2023 at 10:02
  • $\begingroup$ Residuals only work for a subset of regression models, and have other disadvantages noted by others here. I’d recommend either not worrying about collinearity (if you are just predicting Y) or using data reduction before modeling, e.g., variable clustering, redundancy analysis, sparse principal component analysis, as discussed here. $\endgroup$ Commented Nov 5, 2023 at 13:25
  • $\begingroup$ This won't solve the problem at all. There are much better approaches to this well-studied problem. The most general and principled I have found for analyzing the multicollinearity are described in Belsley, Kuh, & Welsch, Regression Diagnostics. That might be a good place to begin. $\endgroup$
    – whuber
    Commented Nov 5, 2023 at 16:44

2 Answers 2

7
$\begingroup$

Note first that standard linear regression only requires that $x$-variables are not perfectly collinear, i.e., the ${\bf X}$-matrix is of lower rank than the number of variables. As @Alex Teush's answer says, in that case you don't win anything; your proposal basically amounts to removing a variable. Otherwise you need to be clear about what problem you actually want to solve. The problem is not with the mathematical solution of the regression problem (as there is no such problem), rather potentially with the interpretation of coefficients and their variation.

Let's say you run a regression $x_2=ax_1+e,$ then $e=x_2-ax_1$. Now LS linear regression is affine equivariant, which implies in particular that if you replace $x$-variables by linear combinations of them (which is what you do here), regression coefficients will change accordingly and the result will be mathematically equivalent. So your proposal doesn't really change the mathematics.

Regarding interpretation there are pros and cons of your proposal. The major issue with interpretation under multicollinearity is that if several variables share the same information, it isn't clear to what variable to attribute a "joint effect". So in the situation with $x_1$ and $x_2$ collinear, the coefficient estimators of both $x_1$ and $x_2$ will have a low precision, because it isn't clear whether the influence of the joint information is due to $x_1$ or $x_2$.

What is changed is the interpretation of the regression coefficients. What your proposal does is it only attributes to the new variable $e$ the information that is not explained by $x_1$. This has the advantage that you can clearly separate the information in $x_1$ from the information in $e$, which clarifies interpretation. However, this advantage comes at a price. If you interpret the coefficient of $x_1$ as its "effect", you are here implicitly assuming that the joint information in $x_1$ and $x_2$ really is only from $x_1$. So if you are interested in the relative importance of the effects of $x_1$ and $x_2$, this cannot be addressed in this way. You basically make the problem of the lack of identifiability of the relative effects of $x_1$ and $x_2$ go away by deciding in advance that only what cannot be explained by $x_1$ is attributed to your new variable $e$. Whether this makes sense depends on background knowledge, particularly on how $x_1$, $x_2$, and $Y$ are causally related. It can be a good idea if indeed $x_1$ is the causal origin of all joint information. It won't help at all if you can't assume this but rather want to find out about it.

In any case note that your idea is maybe not as helpful as you hoped, but neither is there anything really wrong with it, as you will run a regression that is mathematically equivalent.

Also note that the problem of lack of identifiability of the relative contributions of the different variables in case of multicollinearity (or more generally any kind of dependence between them) cannot be solved by any method, as the data do not hold the required information to "solve" this. (Existing approaches such as penalisation can stabilise estimators, which is fine, but doesn't address this issue, unless you make formal use of additional prior information about causal relationships.)

$\endgroup$
1
$\begingroup$

Can't see it working. Since the relationship between $x_1$ and $x_2$ is "almost linear", your new dataset will still suffer from multicollinearity.

One can easily see this, assuming that the multicollinearity is absolute, i.e., that there exists $a\ne0$ such that $x_2=ax_1$. In which case, you're replacing the $x_2$ by a zero column ($e_1 = 0$). I can't see how it benefits you.

There are many ways to tackle multicollinearity: Removing redundant features, adding interactions between features to your model, using transformations (for example $\log(x_1)$), regularization, etc. Sometimes you can even have the model as it is. It depends on what you need the model for\what you want to do with it.

Edit:

  1. By "adding interactions between features" I meant replacing correlating features by their interaction, or at least some of them, in order the link between them would not be linear.
  2. The OP's solution is also discussed in this article, section 5.1. The authors don't recommend using it.
$\endgroup$
3
  • $\begingroup$ can you share a reference backing up that adding interactions can reduce collinearity? $\endgroup$ Commented Nov 5, 2023 at 18:43
  • $\begingroup$ I meant replacing features with high correlation by interaction between features. I edited my answer following you comment. $\endgroup$
    – Alex Teush
    Commented Nov 6, 2023 at 7:18
  • $\begingroup$ I think interaction should be replaced by weighted sum. $\endgroup$ Commented Nov 6, 2023 at 12:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.