# Dealing with multicollinearity when removing a highly collinear predictor reduces significance

In my experience with dealing with multicollinearity, often removing one collinear variable from the model results in the other collinear variable(s) becoming significant (assuming that all the collinear variables are significantly correlated with the dependent variable bivariately.

However, I've recently encountered a situation where I have a model shown below in which $X_2$ and $X_3$ are highly correlated ($r > 0.95$) and the tolerance scores for the two variables are below $0.1$

$$Y = \beta_0\ + \beta_1X_1\ + \beta_2X_2\ + \beta_3X_3\ +\beta_4X_1X_2\ +\beta_5X_1X_3\$$

*all variables are continuous.

The results of the regression model show that all 5 slopes are significant (3 slopes for the main effects and 2 slopes for the interactions). One of the possible solutions is to remove one of the highly collinear predictors. If I remove one of them - say $X_2$, I get a new model as shown below.

$$Y = \beta_0^\prime + \beta_1^\prime X_1\ + \beta_3^\prime X_3\ +\beta_5^\prime X_1X_3\$$

After $X_2$ is removed, while $\beta_0^\prime$, $\beta_1^\prime$, and $\beta_3^\prime$ are significant, crucially, $\beta_5^\prime$, the slope for the interaction term, is no longer significant. The same thing happens if I try to remove $X_3$. So I wonder what may be the reason that causes this pattern to arise and how it can be potentially dealt with. Thank you in advance!

• It's easy to create examples: after $X_2$ is removed, $X_3$ becomes a proxy for it, so if $\beta_4+\beta_5\approx 0$, the coefficient $\beta_5'$ should be close to zero, greatly reducing its p-value. A more fundamental question is why do you think there is a problem in the first place? Collinearity is tolerable in many circumstances, especially when you have lots of data, the condition numbers are small enough to indicate no catastrophic loss of precision, and/or you value a good fit more than a simple model. – whuber Dec 11 '12 at 7:54
• The largest condition index is approximately 6, so I suppose it's not a huge problem? Also, my sample size is about 600. My reaction towards collinearity has to do with all the classes I've taken in which collinearity was stressed to be an issue that needed to be addressed...So whenever collinearity is present, I feel like I need to do something about it :) – BHB Dec 11 '12 at 8:20
• A condition number of 6 with 600 points is not a problem at all. Collinearity should be near the bottom of your list of things to deal with for these data. Higher up on the list would be which variables to include, outliers, influence, heteroscedasticity, goodness-of-fit, interactions, and even higher-order interactions. Monitor collinearity only to make sure results are not affected by numerical instability. – whuber Dec 11 '12 at 8:59

In addition to condition indexes (and developed after Belsley's books were written) in R there is the perturbpackage that examines the problem of colinearity by adding small amounts of random noise to the input data and seeing what happens; one of the problems that colinearity can cause is that small changes to the input data can lead to huge changes in the regression results. In one of Belsley's books he gives an example where changing the data in the third or fourth significant digit reverses the signs of regression coefficients.