Why can variable importance be negative/zero while its correlation with the response variable is high? - Cross Validated most recent 30 from stats.stackexchange.com 2019-06-26T08:01:13Z https://stats.stackexchange.com/feeds/question/332015 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/332015 0 Why can variable importance be negative/zero while its correlation with the response variable is high? Grint https://stats.stackexchange.com/users/172728 2018-03-06T15:48:55Z 2018-03-24T15:08:12Z <p>I don't have a working example for this, as I'm using a large dataset in R with the <em>ranger</em> package (Random Forest algorithm)</p> <p>I fit a model using the <em>ranger</em> package with predictors \$X_1,...,X_k\$ and a response variable \$Y\$ with the purpose of looking at the variable importance of each predictor. After fitting the model, I calculated variable importance using the permutation method and <em>importance()</em>.</p> <p>One of the variables (say \$X_1\$) is highly correlated with the response variable \$Y\$ (~0.7), but based on the Random Forest model the variable importance of \$X_1\$ is negative! I would assume if a variable is highly correlated with the response, it would be seen as more important</p> <p>I'm not sure if there's a simple explanation for this?</p> <p>Thanks so much!</p> https://stats.stackexchange.com/questions/332015/-/336465#336465 1 Answer by jonnor for Why can variable importance be negative/zero while its correlation with the response variable is high? jonnor https://stats.stackexchange.com/users/201327 2018-03-24T15:08:12Z 2018-03-24T15:08:12Z <p>The feature importance is based on the features that were actually used in the decision trees, which is decided on some estimation of information gain (Gini,entropy etc). If the predictors are correlated with eachother, it can be that after splitting on for example \$X_5\$ there is no more information gain to be had from later also splitting on \$X_1\$. In this case the feature importance of \$X_5\$ will be high, and for \$X_1\$ very low or zero.</p> <p>If you believe that \$X_1\$ might actually be a better/preferrable predictor, then leave \$X_5\$ out and run training again.</p>