14
$\begingroup$

Say we have a binary classification problem with mostly categorical features. We use some non-linear model (e.g. XGBoost or Random Forests) to learn it.

  • Should one still be concerned about multi-collinearity? Why?
  • If the answer to the above is true, how should one fight it considering that one is using these types of non-linear models?
$\endgroup$
7
$\begingroup$

Multi-collinearity will not be a problem for certain models. Such as random forest or decision tree. For example, if we have two identical columns, decision tree / random forest will automatically "drop" one column at each split. And the model will still work well.

In addition, regularization is a way to "fix" Multi-collinearity problem. My answer Regularization methods for logistic regression gives details.

| cite | improve this answer | |
$\endgroup$
  • 5
    $\begingroup$ I think this would be improved if you elaborated on exactly what the problem being "fixed" by regularization is. $\endgroup$ – Matthew Drury Mar 8 '17 at 21:47
3
$\begingroup$

Late to the party, but here is my answer anyway, and it is "Yes", one should always be concerned about the collinearity, regardless of the model/method being linear or not, or the main task being prediction or classification.

Assume a number of linearly correlated covariates/features present in the data set and Random Forest as the method. Obviously, random selection per node may pick only (or mostly) collinear features which may/will result in a poor split, and this can happen repeatedly, thus negatively affecting the performance.

Now, the collinear features may be less informative of the outcome than the other (non-collinear) features and as such they should be considered for elimination from the feature set anyway. However, assume that the features are ranked high in the 'feature importance' list produced by RF. As such they would be kept in the data set unnecessarily increasing the dimensionality. So, in practice, I'd always, as an exploratory step (out of many related) check the pairwise association of the features, including linear correlation.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$
  1. Should one still be concerned about multi-collinearity? Why?

If the non-linear model is tree-based model, then you shouldn't consider it serious. Different tree model will have different deal method, such as the random forest will keep them both (because they build the tree independently, and random select the feature for every trees), but it have no effect about the prediction performance, even you remove the redundant one. But for xgboost, it will choose anyone of them, and use it until the last tree build.

  1. If the answer to the above is true, how should one fight it considering that one is using these types of non-linear models?

It just about the interpretation meaning, so remove the highly correlation variable is suggested.

| cite | improve this answer | |
$\endgroup$
-3
$\begingroup$

Multi-collinearity is always a possible problem. Variables that are predictors in the model will affect the prediction when they are linearly related (i.e., when collinearity is present).

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thanks, if (1) the focus is prediction performance (and not the interpretability) and (2) the model is non-linear, would you mind elaborating on why this can still be a problem? (and how exactly it would manifest itself?) $\endgroup$ – Josh Mar 8 '17 at 20:23
  • $\begingroup$ These variables that are predictors in the model will affect the prediction when they are linearly related (i.e. collinearity is present). $\endgroup$ – Michael R. Chernick Mar 8 '17 at 20:26
  • 1
    $\begingroup$ Affect the prediction how, exactly? BTW, stats.stackexchange.com/a/138082/99274, put some links in your answer or face the wrath of the "been there, done that" crowd. $\endgroup$ – Carl Mar 8 '17 at 21:19
  • 7
    $\begingroup$ Since classification is so closely related to prediction, and prediction tends not to suffer from multicollinearity, it is important to support your contention that it's always a "possible problem," especially for the particular models mentioned in the question. What manner of problem would that be for classification and why? $\endgroup$ – whuber Mar 8 '17 at 21:21
  • 12
    $\begingroup$ Im pretty sure you're begging the question. Whuber asked why prediction suffers from multicollinearity, and you basically responded "Prediction suffers from multicollinearity because prediction suffers from multicollinearity." $\endgroup$ – Matthew Drury Mar 8 '17 at 22:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.