Scale of variable importance in randomForest, party & gbm

Question

I've computed some variable measures using the packages, gbm, randomForest and party. I develop binary classification models predicting survival in cancer patients. Although the gbm package, randomForest package and party package is well described in both CRAN files, vignettes and publications, they don't mention much about the x-axis scale or implications, other than that gbm normalizes the scale to 100. I obtained these results from random forest computed with party package (cforest function):

So the x-axes ranges from 0 to 0.03, which seems strange to an amateur like me. I guess these scales are arbitrary in a sense, but should I be worried? Does the scale matter at al?

Answer 1

For the variable importance in the party package help("varimp", package = "party") has a short explanation.

Value: A vector of 'mean decrease in accuracy' importance scores.

More details and the exact formulas can be found in Strobl et al. (2008). "Conditional Variable Importance for Random Forests." BMC Bioinformatics, 9(307). doi:10.1186/1471-2105-9-307

However, the values of these importances may also change with the type of permutation (conditional vs. unconditional), the evaluation set (in-bag vs. out-of-bag), etc. And sometimes the importances are rescaled as you point out. So it is not always possible to exactly compare the numeric values between different packages/functions.

Answer 2

More generally, some comments on the slippery nature of variable importance may be found in Section 12 of the vignette for the earth package. (These comments were written for earth/MARS models, but the comments are mostly generally applicable.)

Correlations (and more generally, any interactions) between variables can mess up estimates of variable importance, as discussed in the Strobl et al. paper (cited in the varimp help page in the party package).

Usually variable importance is relative so, as you mention in your question, the relative scale doesn't matter.

Earth/MARS models can also generate negative importances, and how that happens is described in Section 12.3 of the earth vignette. It's because adding the variable has a deleterious effect on the model, as measured in terms of the model's estimated predictive power on new data.