0
$\begingroup$

I have built two binary classification models, one based on LASSO logistic regression and the second one using Random Forest. Out of 50 variables, 3 are highly correlated with each other. After running LASSO the coefficient of one of these correlated predictors has been shrunk to zero (while two others have relatively high coefficients). For simplicity let's call this variable $X_1$. On the other hand in Random Forest all these three predictors are evaluated as really important according to both, mean accuracy decrease (permutation-based) and mean Gini decrease tests.

If I am not mistaken LASSO tends to exclude some correlated predictors which in theory would explain the difference in variable importance of the models. Also, I guess that the difference may be caused by the non-linear relation between the outcome variable and $X_1$. However, when I run the Random Forest on the data set in which I excluded $X_1$ completely, to my surprise the overall accuracy of the model did not decrease. I have no idea how to explain this behaviour and I would be grateful for any directions.

(If needed I may share the code as well (R) although I do not see any problems in it)

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ If X1 is highly correlated with X2, then removing X1 from the set of predictors in RF will not reduce its fit, as most of the information is still contained in X2. That's also the reason why lasso could shrink the coefficient to 0. X1 has no ADDITIONAL value to those already in the model. $\endgroup$
    – Carsten
    Commented Dec 19, 2019 at 15:36
  • $\begingroup$ Ok but in that case shouldn't permutation test catch it and as a consequence indicate much lower importance for X1? $\endgroup$
    – Tomek
    Commented Dec 19, 2019 at 15:38
  • $\begingroup$ No, permutation importance will not fix this. As long as x1 is used for splitting and substantially reduces training error, as indicated by the decrease in Gini, then permuting the values of x1 will still yield higher OOB MSE, yielding a substantial permutation variable importance. A bagged ensemble is more likely to prefer only one of a set of correlated predictors, and both training error importances and permutation importances will reflect this. $\endgroup$
    – Marjolein Fokkema
    Commented Feb 22, 2021 at 23:28

1 Answer 1

Reset to default
2
$\begingroup$

One hypothesis that seems to fit your explanation is that X1 is correlated with a combination of some of the other variables (i.e. maybe X13, X43, and X44 explain X1). You can check whether it's correlated with the other variables (not just 1 vs 1 correlations) using VIF. This explanation could explain:

(1) X1 shrunk by LASSO, which tends to be an aggressive shrinker (2) X1 deemed important by RF's variable importance metrics (3) RF performing just as well without X1

Of course there are other explanations but this seems to work! Try VIF if you think this could be the explanation.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks Alex, I have checked VIF on the basic glm and the values for X1, X2 and X3 were equal to 2.57, 3.18 and 1.43 respectively. I believe that the obtained values are acceptable. Which other explanations did you have in mind? Moreover, as long as I agree with your 1 and 3 points I'm not sure with the second one. I thought that if X1 was explained by other variables then after permuting X1 in accuracy based test, the accuracy of the model shouldn't drop and therefore the variable should not be indicated as really important. $\endgroup$
    – Tomek
    Commented Dec 20, 2019 at 9:52
  • 1
    $\begingroup$ It is also possible that X1 is a (near-)perfect transformation of one of the other predictor variables. $\endgroup$
    – Marjolein Fokkema
    Commented Feb 25, 2021 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.