In traditional statistics, while building a model, we check for multicollinearity using methods such as estimates of the variance inflation factor (VIF), but in machine learning, we instead use regularization for feature selection and don't seem to check whether features are correlated at all. Why do we do that?


Considering multicollineariy is important in regression analysis because, in extrema, it directly bears on whether or not your coefficients are uniquely identified in the data. In less severe cases, it can still mess with your coefficient estimates; small changes in the data used for estimation may cause wild swings in estimated coefficients. These can be problematic from an inferential standpoint: If two variables are highly correlated, increases in one may be offset by decreases in another so the combined effect is to negate each other. With more than two variables, the effect can be even more subtle, but if the predictions are stable, that is often enough for machine learning applications.

Consider why we regularize in a regression context: We need to constrict the model from being too flexible. Applying the correct amount of regularization will slightly increase the bias for a larger reduction in variance. The classic example of this is adding polynomial terms and interaction effects to a regression: In the degenerate case, the prediction equation will interpolate data points, but probably be terrible when attempting to predict the values of unseen data points. Shrinking those coefficients will likely minimize or entirely eliminate some of those coefficients and improve generalization.

A random forest, however, could be seen to have a regularization parameter through the number of variables sampled at each split: you get better splits the larger the mtry (more features to choose from; some of them are better than others), but that also makes each tree more highly correlated with each other tree, somewhat mitigating the diversifying effect of estimating multiple trees in the first place. This dilemma compels one to find the right balance, usually achieved using cross-validation. Importantly, and in contrast to a regression analysis, no part of the random forest model is harmed by highly collinear variables: even if two of the variables provide the same child node purity, you can just pick one without diminishing the quality of the result.

Likewise, for something like an SVM, you can include more predictors than features because the kernel trick lets you operate solely on the inner product of those feature vectors. Having more features than observations would be a problem in regressions, but the kernel trick means we only estimate a coefficient for each exemplar, while the regularization parameter $C$ reduces the flexibility of the solution -- which is decidedly a good thing, since estimating $N$ parameters for $N$ observations in an unrestricted way will always produce a perfect model on test data -- and we come full circle, back to the ridge/LASSO/elastic net regression scenario where we have the model flexibility constrained as a check against an overly optimistic model. A review of the KKT conditions of the SVM problem reveals that the SVM solution is unique, so we don't have to worry about the identification problems which arose in the regression case.

Finally, consider the actual impact of multicollinearity. It doesn't change the predictive power of the model (at least, on the training data) but it does screw with our coefficient estimates. In most ML applications, we don't care about coefficients themselves, just the loss of our model predictions, so in that sense, checking VIF doesn't actually answer a consequential question. (But if a slight change in the data causes a huge fluctuation in coefficients [a classic symptom of multicollinearity], it may also change predictions, in which case we do care -- but all of this [we hope!] is characterized when we perform cross-validation, which is a part of the modeling process anyway.) A regression is more easily interpreted, but interpretation might not be the most important goal for some tasks.

  • 1
    $\begingroup$ For causal regression modeling, using techniques like propensity scoring or regression adjustment, collinearity can be a problem even for prediction, because usually the goal is to fit a model either exclusively on a control / unexposed group and then estimate outcomes using that model on an experimental group, or else combine the two groups but use an indicator variable to measure the effect, controlling for other factors, of being in the experimental group. $\endgroup$ – ely Jul 31 '18 at 12:24
  • 1
    $\begingroup$ If collinearity creates errors in the coefficients, then the extended regression to the experimental group won't work. Likewise, the coefficient estimate for the indicator variable of having received treatment could be thrown off, if doing a single regression across both sub-samples. Modern machine learning techniques are not usually used to analyze these types of causality problems, and so nobody has had to confront the need for tooling to account for it. $\endgroup$ – ely Jul 31 '18 at 12:25
  • $\begingroup$ @ely, in your first example, colinearity (among the covariates, not the treatment) does not cause a problem, because again the goal is the prediction of the counterfactual outcomes, and colinearity is not a problem with prediction. Also, modern ML methods are frequently used in causal inference; generalized boosted modeling and random forests are widely used to estimate propensity scores, and TMLE uses ML methods to impute the counterfactual outcomes. I would argue a strength of causal methods is that colinearity is not usually a problem for them. $\endgroup$ – Noah Aug 3 '18 at 17:25
  • $\begingroup$ @Noah Usually it is the interpretation of the exposure coefficient that matters (and interpretation of the other effect estimates too), and not solely raw prediction accuracy. I realize my comment didn't make this clear, but that's why it is an issue. If the overall prediction is good, but is not driven by truly being related to the coefficient estimated for exposure, it's usually an undesirable model for causal inference. $\endgroup$ – ely Aug 3 '18 at 18:06

The reason is because the goals of "traditional statistics" are different from many Machine Learning techniques.

By "traditional statistics", I assume you mean regression and its variants. In regression, we are trying to understand the impact the independent variables have on the dependent variable. If there is strong multicollinearity, this is simply not possible. No algorithm is going to fix this. If studiousness is correlated with class attendance and grades, we cannot know what is truly causing the grades to go up - attendance or studiousness.

However, in Machine Learning techniques that focus on predictive accuracy, all we care about is how we can use a set of variables to predict another set. We don't care about the impact these variables have on each other.

Basically, the fact that we don't check for multicollinearity in Machine Learning techniques isn't a consequence of the algorithm, it's a consequence of the goal. You can see this by noticing that strong collinearity between variables doesn't hurt the predictive accuracy of regression methods.


There appears to be an underlying assumption here that not checking for collinearity is a reasonable or even best practice. This seems flawed. For example, checking for perfect collinearity in a dataset with many predictors will reveal whether two variables are actually the same thing e.g. birth date and age (example taken from Dormann et al. (2013), Ecography, 36, 1, pp 27–46). I have also sometimes seen the issue of perfectly correlated predictors arise in Kaggle competitions where competitors on the forum attempt to eliminate potential predictors which have been anonymised (i.e. the predictor label is hidden, a common problem in Kaggle and Kaggle-like competitions).

There is also still an activity in machine learning of selecting predictors - identifying highly correlated predictors may allow the worker to find predictors which are proxies for another underlying (hidden) variable and ultimately find one variable which does the best job of representing the latent variable or alternatively suggest variables which may be combined (e.g. via PCA).

Hence, I would suggest that although machine learning methods have usually (or at least often) been designed to be robust in the face of correlated predictors, understanding the degree to which predictors are correlated is often a useful step in producing a robust and accurate model, and is a useful aid for obtaining an optimised model.


The main issue with multicollinearity is that it messes up the coefficients (betas) of independent variables. That's why it's a serious issue when you're studying the relationships between variables, establishing causality etc.

However, if you're not interested in understanding the phenomenon so much, but are solely focused on prediction and forecasting, then multicollinearity is less of an issue. Or at least that's what people think about it.

I'm not talking about perfect multicollinearity here, which is a technical issue or identification issue. Technically, it simply means that the design matrix leads to singularity, and the solution is not defined.

  • 4
    $\begingroup$ Even with perfect collinearity, the predictions are well defined. $\endgroup$ – whuber Aug 25 '15 at 3:03
  • $\begingroup$ @whuber, if you use OLS, the stat package is likely to throw an error, because it wont be able to invert the matrix. The smart ones may drop one of the independent vars, and move on though. $\endgroup$ – Aksakal Aug 25 '15 at 3:06
  • 2
    $\begingroup$ If you use generalized inverse then this singularity is not an problem. $\endgroup$ – Analyst Aug 25 '15 at 4:42
  • 1
    $\begingroup$ I don't follow your logic, Aksakal: are you trying to suggest that machine learning techniques differ from statistical techniques in that the former somehow don't have problems with reduced-rank matrices? It's an interesting idea to explore. $\endgroup$ – whuber Aug 25 '15 at 12:40
  • $\begingroup$ @whuber, a perfect multicollinerity (reduced rank design matrix) is better be dealt with, in my opinion. If you have the same column twice in the matrix it's simply not cool, but less than perfect multicollinearity is not a big issue in some cases. It's not even in machine learning but also in forecasting. By dropping a column, I meant that the software will remove the variable from the specification to reduce the dimension of the matrix to match its rank. $\endgroup$ – Aksakal Aug 25 '15 at 12:51

The regularization in those machine learning stabilizes the regression coefficients, so at least that effect of multicollinearity tamed. But more importantly, if you're going for prediction (which machine learners often are), then the multicollinearity "problem" wasn't that big of a problem in the first place. It's a problem when you need to estimate a particular coefficient and you don't have the information.

Also, my answer to "When does LASSO select correlated predictors" might be helpful to you.


I think that multicollinearity should be checked in machine learning. Here is why: Suppose that you have two highly correlated features X and Y in our dataset. This means that the response plane is not reliable (a small change in the data can have drastic effects on the orientation of the response plane). Which implies that the predictions of the model for data points far away from the line, where X and Y tend to fall, are not reliable. If you use your model for predictions for such points the predictions probably will be very bad. To put it in other words, when you have two highly correlated features, as a model, you are learning a plane where actually the data mostly falls in a line. So, it is important to remove highly correlated features from your data for preventing unreliable models and erroneous predictions.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.