0
$\begingroup$

I don't have a working example for this, as I'm using a large dataset in R with the ranger package (Random Forest algorithm)

I fit a model using the ranger package with predictors $X_1,...,X_k$ and a response variable $Y$ with the purpose of looking at the variable importance of each predictor. After fitting the model, I calculated variable importance using the permutation method and importance().

One of the variables (say $X_1$) is highly correlated with the response variable $Y$ (~0.7), but based on the Random Forest model the variable importance of $X_1$ is negative! I would assume if a variable is highly correlated with the response, it would be seen as more important

I'm not sure if there's a simple explanation for this?

Thanks so much!

Improve this question
$\endgroup$
2
  • $\begingroup$ IMHO, variable importance for random forest algorithm is 0 or positive. It cannot be negative. Is it possible that something is wrong with your code? $\endgroup$
    – lanenok
    Mar 6, 2018 at 18:57
  • $\begingroup$ Negative variable importances are perfectly possible for permutation importances. This would mean that the error estimate (e.g., MSE) was higher when using the original predictor variable values, than when using the permuted values. If a variable was hardly predictive of the outcome, but still selected for some of the splits, randomly permuting the values of that variable may send some observations down a path in the tree which happens to yield a more accurate predicted value, than the path and predicted value that would have been obtained with the original ordering of the variable. $\endgroup$
    – Marjolein Fokkema
    Feb 23, 2021 at 23:49

1 Answer 1

Reset to default
1
$\begingroup$

The feature importance is based on the features that were actually used in the decision trees, which is decided on some estimation of information gain (Gini,entropy etc). If the predictors are correlated with eachother, it can be that after splitting on for example $X_5$ there is no more information gain to be had from later also splitting on $X_1$. In this case the feature importance of $X_5$ will be high, and for $X_1$ very low or zero.

If you believe that $X_1$ might actually be a better/preferrable predictor, then leave $X_5$ out and run training again.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.