Say I have the following multiple linear regression:

Y ~ X1 + X2 + X3 + X4

All X variables are independent, but X1 and X2 look kind of linearly related when plotted against one another (e.g., R-squared ~0.5, if I fit a simple linear regression to X1 ~ X2). Will BIC or AIC throw out one of the variables because this? How much collinearity is too much? Is there a formal procedure for identifying and dealing with such collinearity?


A1: We consider AIC or BIC to compare models within a set of good candidate models. By saying a good candidate models I mean the models without any major problem ... that includes multicollinearity as well. So you need to first come up with some good models and then try to compare them using AIC or BIC.

A2 & A3: There is no unique answer to this question. Most of the time we will look at Variance inflation factor i.e. $VIF_j=\dfrac{1}{1-R^2_{j}}$, where $R^2_{j}$ is the R-squared of the model with $X_j$ as the response over all other remaining independent variables. As a rule of thumb, if the $VIF>10$ then there is a serious multicollinearity problem. Although, in some text books they consider cut-off value of 5.

  • $\begingroup$ Thanks for the helpful reply, Stat. I used VIF to help me identify collinearity amongst the variables and then updated the model accordingly. $\endgroup$ – Lyngbakr Mar 4 '15 at 19:01
  • $\begingroup$ Does it have to be so restrictive? I doubt that the theory of AIC and BIC requires "good" models. And multicollinearity is not a feature of a model but rather of a particular sample (since there is no population counterpart of multicollinearity). One of my statistics professors used to say that one should not bother about multicollinearity precisely because AIC or BIC will take care of it (i.e. will select some other model than a model with high multicollinearity). $\endgroup$ – Richard Hardy Mar 4 '15 at 19:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.