3
$\begingroup$

In my experience with dealing with multicollinearity, often removing one collinear variable from the model results in the other collinear variable(s) becoming significant (assuming that all the collinear variables are significantly correlated with the dependent variable bivariately.

However, I've recently encountered a situation where I have a model shown below in which $X_2$ and $X_3$ are highly correlated ($r > 0.95$) and the tolerance scores for the two variables are below $0.1$

$$ Y = \beta_0\ + \beta_1X_1\ + \beta_2X_2\ + \beta_3X_3\ +\beta_4X_1X_2\ +\beta_5X_1X_3\ $$

*all variables are continuous.

The results of the regression model show that all 5 slopes are significant (3 slopes for the main effects and 2 slopes for the interactions). One of the possible solutions is to remove one of the highly collinear predictors. If I remove one of them - say $X_2$, I get a new model as shown below.

$$ Y = \beta_0^\prime + \beta_1^\prime X_1\ + \beta_3^\prime X_3\ +\beta_5^\prime X_1X_3\ $$

After $X_2$ is removed, while $\beta_0^\prime$, $\beta_1^\prime$, and $\beta_3^\prime$ are significant, crucially, $\beta_5^\prime$, the slope for the interaction term, is no longer significant. The same thing happens if I try to remove $X_3$. So I wonder what may be the reason that causes this pattern to arise and how it can be potentially dealt with. Thank you in advance!

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ It's easy to create examples: after $X_2$ is removed, $X_3$ becomes a proxy for it, so if $\beta_4+\beta_5\approx 0$, the coefficient $\beta_5'$ should be close to zero, greatly reducing its p-value. A more fundamental question is why do you think there is a problem in the first place? Collinearity is tolerable in many circumstances, especially when you have lots of data, the condition numbers are small enough to indicate no catastrophic loss of precision, and/or you value a good fit more than a simple model. $\endgroup$
    – whuber ♦
    Commented Dec 11, 2012 at 7:54
  • $\begingroup$ The largest condition index is approximately 6, so I suppose it's not a huge problem? Also, my sample size is about 600. My reaction towards collinearity has to do with all the classes I've taken in which collinearity was stressed to be an issue that needed to be addressed...So whenever collinearity is present, I feel like I need to do something about it :) $\endgroup$
    – BHB
    Commented Dec 11, 2012 at 8:20
  • 4
    $\begingroup$ A condition number of 6 with 600 points is not a problem at all. Collinearity should be near the bottom of your list of things to deal with for these data. Higher up on the list would be which variables to include, outliers, influence, heteroscedasticity, goodness-of-fit, interactions, and even higher-order interactions. Monitor collinearity only to make sure results are not affected by numerical instability. $\endgroup$
    – whuber ♦
    Commented Dec 11, 2012 at 8:59

2 Answers 2

Reset to default
1
$\begingroup$

Correlation between two independent variables is not necessarily a sign of troublesome collinearity. The guru of collinearity, David Belsley, has shown this in his books: Conditioning Diagnostics: Collinearity and Weak Data in Regression and Regression Diagnostics: Identifying Influential Points and Sources of Collinearity.

In the comments, @Whuber points out that collinearity is not always a problem that has to be dealt with and that your maximum condition index indicates that, here, it is not even a problem at all.

At the other end, it is also possible to have very high collinearity without any high correlations. One example of this is if there are 10 IVs, 9 of which are independent and the 10th is the sum of the other 9.

In addition to condition indexes (and developed after Belsley's books were written) in R there is the perturb package that examines the problem of collinearity by adding small amounts of random noise to the input data and seeing what happens; one of the problems that collinearity can cause is that small changes to the input data can lead to huge changes in the regression results. In one of Belsley's books he gives an example where changing the data in the third or fourth significant digit reverses the signs of regression coefficients.

Improve this answer
$\endgroup$
-2
$\begingroup$

In case of collinearity between two variables and you have to choose which one to keep is by calculating the correlation of those two predictors with all other predictors and then taking averages of all those correlation and then keep one which has lower correlation with other predictors. To remove multicollinearity, you can do this for all predictors by setting some threshold to keep.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.