I know that wikipedia references are sometimes frowned upon here, but this one has me puzzled: Wikipedia - Multicollinearity

I know what multicollinearity is, and today I tried figuring out how/if it would affect performance of machine learning models.

At the beginning of the article, it says

Multicollinearity does not reduce the predictive power or reliability of the model as a whole

...but, as I read on it says that

A principal danger of such data redundancy is that of overfitting in regression analysis models

and I know that overfitting increases variance greatly, and can degrade performance severely.

Are either of these or both of these statements wrong?

  • 2
    $\begingroup$ Multicollinearity does not increase bias, but it increases variance (overfitting). $\endgroup$ – William Chiu Jan 10 '16 at 18:00
  • $\begingroup$ So, are both of the above statements still true? And, is there a better way of explaining this? $\endgroup$ – Candic3 Jan 10 '16 at 18:02

There is an important qualifier in the continuation of the first cited quote from Wikipedia: "Multicollinearity does not reduce the predictive power or reliability of the model as a whole, at least within the sample data set" (emphasis added).

Unless there is a singular design matrix, multicollinearity does not prevent fitting a model to an individual data sample. I take that to be the point of the first Wikipedia quote. A regression model can capture the data sample well, including all of the peculiarities and noise of the particular data sample, in the presence of multicollinearity.

The problems arise when you try to apply the model outside the original data set to the underlying population. Multicollinearity will severely affect performance outside of the original data sample, as you recognize.

  • 1
    $\begingroup$ Thank you. So, I suppose by "sample data set" we mean training data set? And then, the fact that the regression model can capture the noise, as you suggest, would mean that the statement cautioning against overfitting has a lot of validity $\endgroup$ – Candic3 Jan 11 '16 at 2:32
  • $\begingroup$ Any response on my first comment? anyway....I suppose some kind of cross-validation would uncover this problem, because the variance from each training/testing split would cause a lot of MSE. $\endgroup$ – Candic3 Aug 26 '16 at 6:30
  • 1
    $\begingroup$ @Candic3 : Yes, the "sample data set" would be the particular sample used for building the model, the training set if you use a training/test set approach. My favorite way to display the problem with multicollinearity is to repeat model building on a few hundred bootstrap samples of the original data. Then see how variable the regression coefficients are among those models, or how poorly those models fit the original data set. Ridge regression can combine information usefully from multicollinear predictors, minimizing overfitting and providing more reliable predictions on new data. $\endgroup$ – EdM Aug 26 '16 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.