Why Machine Learning is indifferent regarding the dependence / muti-collinearity of predictors in regression?

Question

Machine Learning is in generally impervious to the violation of the independence assumption among the predictors of a regression model -- frequently employing in its models features that are correlated and dependent.

Prediction accuracy empirically does not seem to suffer because of this tolerance to multi-collinearity of predictors.

Why is this the case? After all --speaking for example about logistic regression-- the conditional probability of the even of interest Y to happen given the data X = {X1, X2, ..., Xn} is defined as the product of the conditional probabilites Y | Xi, which is true only if Xi are independent and therefore uncorrelated.

Your advice will be appreciated.

Answer 1

The only problem that collinearity causes is in figuring out exactly which effect each predictor has and whether there is a clear evidence that a predictor matters. There is really no assumption of independence of predictors for regression, unless you want to do these aforementioned things. The conditional probability of the event of interest in logistic regression is also only the product of the conditional probabilites given one predictor, if the predictors are independent. It is not in general defined in this way. And to be honest, unless we are in an experimental setting, where you can manipulate each predictor independently from each other, predictors will in practice never truly be independent.

As soon as you do not care about these things and do only care about prediction, then collinearity becomes somewhat irrelevant (whether we are in a machine learning, statistical or any other framework).

Answer 2

I'd say that it's only about perspective. In statistics you usually check for some assumptions in order to be sure that your model is going to give you good predictions in the data range. Also you choose which method you're going to use in your model .

In machine learning all of these assumptions and checkings are more "empirical" in the sense that for example you could be checking you prediction accuracy after adding some feature engineering that perhaps include feature selection o even doing PCA over the X, so you 'll have independent variables, for example, if this is giving you the best accuracy over your test set.

I think both are valid strategies and it only depends on several aspects:

• your knowledge about the data itself
• the amount of data you have and how good testing you can do with test data
• how many variables are there, 10? , 150? 900? Depending on this it would be more or less reasonable checking assumptions or doing "brute force" engeneering, etc