I have built a logistic regression model with two or more highly correlated factors. I did this by doing a bagging procedure.

In my understanding having highly correlated factors in a prediction model is not an issue (especially after bagging). But in a model where you are trying to explain the target, this could be an issue.

I remember something like the above was stated in a machine book but I can't find it. Do you guys agree with this viewpoint and can point me to a reference or two on this?


You can look at Kutner et al., p. 283:

"The fact that some or all predictor variables are correlated among themselves does not, in general, inhibit our ability to obtain a good fit nor does it tend to affect inferences about mean responses or predicions of new observations, provided these inferences are made within the region of observations."


"The common interpretation of a regression coefficient as measuring the change in the expected value of the response variable [...] while all other predictor variables are held constant is not fully applicable [...] For example, in a regression model for predicting crop yield from amount of rainfall and hours of sunshine, the relation between the two predictor variables makes it unrealistic to consider varying one while holding the other constant."


They can be.

Broadly speaking, highly correlated factors have the following implications:

  • They increase the standard error of estimates of the $\beta$’s
  • They often lead to confusing and misleading results. (see page 2 of this overview for more explanation).
  • If interest is only in estimation and prediction, high correlation can be ignored since it does not affect $\hat{y}$ or its standard error (either $\hat{\sigma}_\hat{y}$ or $\hat{\sigma}_{y-\hat{y}}$).

Since the standard errors of your estimated coefficients are higher, you have less certainty about your predictions. In this sense, they are harder to explain.

Additional dangers are listed on the wikipedia page on multicollinearity. Here, it is pointed out that correlation between the factors can lead to overfitting, and if omitted variable bias is present, correlation between variables can magnify the problem. When a lot of extrapolation is performed, this can lead to large prediction errors.

Sources are given at the bottom of the wikipedia page.

  • $\begingroup$ But I use bagging already. So the SE of the estimates is not that big a concern? $\endgroup$ – xiaodai May 15 '14 at 8:19
  • $\begingroup$ Bagging can improve the standard errors, but I am not sure it completely deals with multicollinearity (perhaps it can help: here they indicate it can help with cases of many variables, but do not explicitly touch upon multicolliearity or highly correlated predictors). Other places I checked, like here seem to know bagging, but advocate other approaches for collinearity. $\endgroup$ – Deathkill14 May 15 '14 at 8:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.