Why is multicollinearity not checked in modern statistics/machine learning

Question

In traditional statistics, while building a model, we check for multicollinearity using methods such as estimates of the variance inflation factor (VIF), but in machine learning, we instead use regularization for feature selection and don't seem to check whether features are correlated at all. Why do we do that?

Answer 1

Considering multicollineariy is important in regression analysis because, in extrema, it directly bears on whether or not your coefficients are uniquely identified in the data. In less severe cases, it can still mess with your coefficient estimates; small changes in the data used for estimation may cause wild swings in estimated coefficients. These can be problematic from an inferential standpoint: If two variables are highly correlated, increases in one may be offset by decreases in another so the combined effect is to negate each other. With more than two variables, the effect can be even more subtle, but if the predictions are stable, that is often enough for machine learning applications.

Consider why we regularize in a regression context: We need to constrict the model from being too flexible. Applying the correct amount of regularization will slightly increase the bias for a larger reduction in variance. The classic example of this is adding polynomial terms and interaction effects to a regression: In the degenerate case, the prediction equation will interpolate data points, but probably be terrible when attempting to predict the values of unseen data points. Shrinking those coefficients will likely minimize or entirely eliminate some of those coefficients and improve generalization.

A random forest, however, could be seen to have a regularization parameter through the number of variables sampled at each split: you get better splits the larger the mtry (more features to choose from; some of them are better than others), but that also makes each tree more highly correlated with each other tree, somewhat mitigating the diversifying effect of estimating multiple trees in the first place. This dilemma compels one to find the right balance, usually achieved using cross-validation. Importantly, and in contrast to a regression analysis, the predictions of the random forest model are not harmed by highly collinear variables: even if two of the variables provide the same child node purity, you can just pick one.

Likewise, for something like an SVM, you can include more predictors than features because the kernel trick lets you operate solely on the inner product of those feature vectors. Having more features than observations would be a problem in regressions, but the kernel trick means we only estimate a coefficient for each exemplar, while the regularization parameter $C$ reduces the flexibility of the solution -- which is decidedly a good thing, since estimating $N$ parameters for $N$ observations in an unrestricted way will always produce a perfect model on test data -- and we come full circle, back to the ridge/LASSO/elastic net regression scenario where we have the model flexibility constrained as a check against an overly optimistic model. A review of the KKT conditions of the SVM problem reveals that the SVM solution is unique, so we don't have to worry about the identification problems which arose in the regression case.

Finally, consider the actual impact of multicollinearity. It doesn't change the predictive power of the model (at least, on the training data) but it does screw with our coefficient estimates. In most ML applications, we don't care about coefficients themselves, just the loss of our model predictions, so in that sense, checking VIF doesn't actually answer a consequential question. (But if a slight change in the data causes a huge fluctuation in coefficients [a classic symptom of multicollinearity], it may also change predictions, in which case we do care -- but all of this [we hope!] is characterized when we perform cross-validation, which is a part of the modeling process anyway.) A regression is more easily interpreted, but interpretation might not be the most important goal for some tasks.

Answer 2

The reason is because the goals of "traditional statistics" are different from many Machine Learning techniques.

By "traditional statistics", I assume you mean regression and its variants. In regression, we are trying to understand the impact the independent variables have on the dependent variable. If there is strong multicollinearity, this is simply not possible. No algorithm is going to fix this. If studiousness is correlated with class attendance and grades, we cannot know what is truly causing the grades to go up - attendance or studiousness.

However, in Machine Learning techniques that focus on predictive accuracy, all we care about is how we can use a set of variables to predict another set. We don't care about the impact these variables have on each other.

Basically, the fact that we don't check for multicollinearity in Machine Learning techniques isn't a consequence of the algorithm, it's a consequence of the goal. You can see this by noticing that strong collinearity between variables doesn't hurt the predictive accuracy of regression methods.

Answer 3

There appears to be an underlying assumption here that not checking for collinearity is a reasonable or even best practice. This seems flawed. For example, checking for perfect collinearity in a dataset with many predictors will reveal whether two variables are actually the same thing e.g. birth date and age (example taken from Dormann et al. (2013), Ecography, 36, 1, pp 27–46). I have also sometimes seen the issue of perfectly correlated predictors arise in Kaggle competitions where competitors on the forum attempt to eliminate potential predictors which have been anonymised (i.e. the predictor label is hidden, a common problem in Kaggle and Kaggle-like competitions).

There is also still an activity in machine learning of selecting predictors - identifying highly correlated predictors may allow the worker to find predictors which are proxies for another underlying (hidden) variable and ultimately find one variable which does the best job of representing the latent variable or alternatively suggest variables which may be combined (e.g. via PCA).

Hence, I would suggest that although machine learning methods have usually (or at least often) been designed to be robust in the face of correlated predictors, understanding the degree to which predictors are correlated is often a useful step in producing a robust and accurate model, and is a useful aid for obtaining an optimised model.

Answer 4

The main issue with multicollinearity is that it messes up the coefficients (betas) of independent variables. That's why it's a serious issue when you're studying the relationships between variables, establishing causality etc.

However, if you're not interested in understanding the phenomenon so much, but are solely focused on prediction and forecasting, then multicollinearity is less of an issue. Or at least that's what people think about it.

I'm not talking about perfect multicollinearity here, which is a technical issue or identification issue. Technically, it simply means that the design matrix leads to singularity, and the solution is not defined.

Answer 5

The regularization in those machine learning stabilizes the regression coefficients, so at least that effect of multicollinearity tamed. But more importantly, if you're going for prediction (which machine learners often are), then the multicollinearity "problem" wasn't that big of a problem in the first place. It's a problem when you need to estimate a particular coefficient and you don't have the information.

Also, my answer to "When does LASSO select correlated predictors" might be helpful to you.

Answer 6

I think that multicollinearity should be checked in machine learning. Here is why: Suppose that you have two highly correlated features X and Y in our dataset. This means that the response plane is not reliable (a small change in the data can have drastic effects on the orientation of the response plane). Which implies that the predictions of the model for data points far away from the line, where X and Y tend to fall, are not reliable. If you use your model for predictions for such points the predictions probably will be very bad. To put it in other words, when you have two highly correlated features, as a model, you are learning a plane where actually the data mostly falls in a line. So, it is important to remove highly correlated features from your data for preventing unreliable models and erroneous predictions.