Say I have the following multiple linear regression:
Y ~ X1 + X2 + X3 + X4
All X variables are independent, but X1 and X2 look kind of linearly related when plotted against one another (e.g., R-squared ~0.5, if I fit a simple linear regression to X1 ~ X2). Will BIC or AIC throw out one of the variables because this? How much collinearity is too much? Is there a formal procedure for identifying and dealing with such collinearity?
A1: We consider AIC or BIC to compare models within a set of good candidate models. By saying a good candidate models I mean the models without any major problem ... that includes multicollinearity as well. So you need to first come up with some good models and then try to compare them using AIC or BIC.
A2 & A3: There is no unique answer to this question. Most of the time we will look at Variance inflation factor i.e. $VIF_j=\dfrac{1}{1-R^2_{j}}$, where $R^2_{j}$ is the R-squared of the model with $X_j$ as the response over all other remaining independent variables. As a rule of thumb, if the $VIF>10$ then there is a serious multicollinearity problem. Although, in some text books they consider cut-off value of 5.
