The Gauss-Markov Assumptions:

  • MLR.1: Linearity in parameters.
  • MLR.2: Random sampling.
  • MLR.3: No perfect multicollinearity.
  • MLR.4: Zero conditional mean

Hence, why should I check for high (but not perfect) collinearity in a linear regression with for example the VIF?


Typically, the regression assumptions are: 1) mean error of zero 2) conditional homoskedasticity 3) error independence 4) normality of the error distribution

I've done some econometrics course work so I am aware how the Gauss-Markov items mention things a bit differently and add two assumptions.

Technically, absence of perfect collinearity isn't a regression assumption, but it assures unique estimates and works better for the matrix algebra because perfect collinearity would cause redundancy in the matrix and makes the matrix non-invertible (singular), which means the determinant is zero (you can't find unique solutions-- many values for the coefficient estimates could work). Avoiding perfect collinearity allows for the case of unique solutions (coefficient estimates).

If you want to make inferences on the estimated slope coefficients, multicollinearity (at a problematic level) can cause inappropriate and misguided inferences such as concluding the wrong magnitude, wrong direction, or both regarding a partial relationship of some Xi with Y. Multicollinearity, by it's nature can introduce instability in the regression parameter estimates, but the model is still unbiased and consistent, all else constant.

If you want to use a model for predictions, though, multicollinearity is not as much of a concern, generally speaking. This is because the predictions remain unbiased (as alluded to above).


When you have approximate colinearity:

  • small changes in the data can lead to big changes in the parameter estimates and
  • the parameter estimates will have huge standard errors

Indeed, Belsely shows examples where a change in the 3rd significant figures of the data makes parameters that were significant flip sign and still be significant - that is, you would make the opposite conclusion about the same parameter.


Your model can still be fitted with colinear features, there is no "a priori" problem with that.

The issue begins when you try to simplify to determine which predictors are having the most influence in the output (this would help you in feature selection for a simpler model), as in this case the model will simply not know which of the colinear variables is really the cause of output changes.

Also, introducing a new highly colinear feature will not improve your model at all, since it will give no more information. A new parameter will have to be estimated, so the model could end up losing quality (variables that used to be "significant" may not be anymore)

Finally, the fact that two variables are correlated in the training set does not mean that will still be the case when applying the model in practice. As the model gets confused the correlated variables, predictions will be inaccurate in this scenario.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.