Why is multicollinearity so bad for machine learning models?

Is there ever a time when we can ignore multicollinearity?

How does regularization ($L_1$, $L_2$) help us deal with multicollinearity?


Multicollinearity simply imlies that one or more of the features in your dataset are useless to the model. Thus you get all the problems associated with more features (i.e. curse of dimensionality), but none of the benefits (e.g. making the classes easier separable).

Many ML algorithms are impervious to problems of this nature. Algorithms that internally perform any form of feature selection and are good with high dimensional data (e.g. tree-based algorithms, lasso) are robust against multicollinearity.

$L_1$ regularization mainly helps models as it provides sparse solutions, robust against multicollinearity. $L_2$ doesn't help as much. Read this article if you are interested on the differences of the two.

As a final note, multicollinearity isn't as big a problem in Machine Learning as you make it out to be. That being said, if such a problem is detected it is almost always beneficial to perform some sort of feature selection, or even PCA to help decorrelate the features.


The easiest way to understand is to imagine that you have two identical features, e.g. temperature in Celsius and Fehrenheits. This is a case of perfect collinearity.

Two things will happen, both bad. One is that at the very least you're going to waste some neurons. In the first layer you have $a^{[0]}_i$ inputs for $i=1,2,\dots,n$ features. Two of these features are essentially the same, but the model doesn't know it and assigns the weights $w_{ij}$ to them:$z_j=\sum_ia^{[0]}_i w_{ij}$, where $j=1,2,\dots,k^{[1]}$ neurons in the first layer. So, you wasted $k^{[1]}$ neurons right out of the gate.

The second thing is that this will degenerate the potential optimal solution. Again, consider the first layer's connections $a^{[0]}_1 w_{1j}+a^{[0]}_2 w_{2j}$, if the first and the second input are collinear, then there's infinite number of combinations that would produce the same exact result as this sumproduct. This is going to be confusing your optimizer, and make its work a little harder.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.