0
$\begingroup$

I don't have a working example for this, as I'm using a large dataset in R with the ranger package (Random Forest algorithm)

I fit a model using the ranger package with predictors $X_1,...,X_k$ and a response variable $Y$ with the purpose of looking at the variable importance of each predictor. After fitting the model, I calculated variable importance using the permutation method and importance().

One of the variables (say $X_1$) is highly correlated with the response variable $Y$ (~0.7), but based on the Random Forest model the variable importance of $X_1$ is negative! I would assume if a variable is highly correlated with the response, it would be seen as more important

I'm not sure if there's a simple explanation for this?

Thanks so much!

$\endgroup$
  • $\begingroup$ IMHO, variable importance for random forest algorithm is 0 or positive. It cannot be negative. Is it possible that something is wrong with your code? $\endgroup$ – lanenok Mar 6 '18 at 18:57
1
$\begingroup$

The feature importance is based on the features that were actually used in the decision trees, which is decided on some estimation of information gain (Gini,entropy etc). If the predictors are correlated with eachother, it can be that after splitting on for example $X_5$ there is no more information gain to be had from later also splitting on $X_1$. In this case the feature importance of $X_5$ will be high, and for $X_1$ very low or zero.

If you believe that $X_1$ might actually be a better/preferrable predictor, then leave $X_5$ out and run training again.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.