Solutions for multicolinearity

Question

how do you linearly combine predictors, such as adding them together to avoid multicollinearity? I have multicolinearity in my regression. say the function is lm(y ~ x + z) where x and z have collinearity. Is one solution to assess the individual regression mod1 = y ~ x and mod2= y ~ z then combine them. How would the combination work? how do you add these two together is it y = the coef x from mod1 + coef of z from mod2?

Answer 1

The usual goal in regression is to include all relevant predictors in the model. Omitting important predictors correlated with included predictors can lead to reversed signs of regression coefficients in standard linear regression (see Simpson's paradox). In logistic or Cox survival regressions, omitting a predictor related to outcome will even bias the coefficients of predictors with which it is not correlated. So starting with separate models for each of x and z is not a good way to proceed.

Multicollinearity (perhaps an over-statement in this case with only 2 predictors) gets a lot of attention but doesn't by itself necessarily lead to problems unless it's so extreme that you can't get reliable estimates of the coefficients. As this answer notes:

... the effects of collinearity are seen in the variances of the parameter estimates, not in the parameter estimates themselves.

The VIF, after all, is the Variance Inflation Factor for a coefficient.

What you are seeing is what you would expect if x and z are correlated with each other and if one or both are directly related to y. Then each of x and z could be significantly related to y individually (based on some p-value cutoff), but including both in a 2-predictor regression could lead to 1 of them appearing to be "not significant." It's quite possible in your 2-predictor model that x is still related to y in the same direction as it was individually, but that a high standard error for its regression coefficient (due to its correlation with z) made it fail to meet some p-value cutoff.

If your intent is prediction of y in new cases based on values of x and z, then there is no reason to omit either of x or z. In such a situation it's best to try to retain all variables related to outcome, even those that don't pass some particular test of statistical significance. Variable selection based on p-values or other relations to outcome "only makes things worse." You do have to take precautions against overfitting or use other modeling approaches when the ratio of cases to predictors gets low, but with only 2 predictors that shouldn't be a problem provided you have more than a couple of dozen cases.