0
$\begingroup$

I've computed some variable measures using the packages, gbm, randomForest and party. I develop binary classification models predicting survival in cancer patients. Although the gbm package, randomForest package and party package is well described in both CRAN files, vignettes and publications, they don't mention much about the x-axis scale or implications, other than that gbm normalizes the scale to 100. I obtained these results from random forest computed with party package (cforest function): enter image description here

So the x-axes ranges from 0 to 0.03, which seems strange to an amateur like me. I guess these scales are arbitrary in a sense, but should I be worried? Does the scale matter at al?

$\endgroup$
3
$\begingroup$

For the variable importance in the party package help("varimp", package = "party") has a short explanation.

Value: A vector of 'mean decrease in accuracy' importance scores.

More details and the exact formulas can be found in Strobl et al. (2008). "Conditional Variable Importance for Random Forests." BMC Bioinformatics, 9(307). doi:10.1186/1471-2105-9-307

However, the values of these importances may also change with the type of permutation (conditional vs. unconditional), the evaluation set (in-bag vs. out-of-bag), etc. And sometimes the importances are rescaled as you point out. So it is not always possible to exactly compare the numeric values between different packages/functions.

$\endgroup$
  • $\begingroup$ thanks for a superb package. For us who are into medical research, this package is a game changer, particularly now that big data is moving into medical research. Variable importance is at the core of clinical research. So, thank you guys for this package. I have some quick questions: (1) despite using 1500 trees and 4000 patients, some variables get negative importance. (2) are you guys planing on making calculations of conditional importance faster? its currently extremely slow, which may of course have a good explanation. $\endgroup$ – Jennifer Mente Feb 24 '18 at 23:23
  • $\begingroup$ Glad if it is useful for you. (1) Negative importances are just the result of random variation and not unusual. (2) We are working on a flexible reimplementation of a flexible random forest algorithm in package partykit. This will also include on overhaul of the variable importance computations. I'm not sure though how much speed-up Torsten was able to attain for conditional importances. Especially for large forests with large trees it is simply burdensome to obtain the conditional permutations. $\endgroup$ – Achim Zeileis Feb 24 '18 at 23:43
1
$\begingroup$

More generally, some comments on the slippery nature of variable importance may be found in Section 12 of the vignette for the earth package. (These comments were written for earth/MARS models, but the comments are mostly generally applicable.)

Correlations (and more generally, any interactions) between variables can mess up estimates of variable importance, as discussed in the Strobl et al. paper (cited in the varimp help page in the party package).

Usually variable importance is relative so, as you mention in your question, the relative scale doesn't matter.

Earth/MARS models can also generate negative importances, and how that happens is described in Section 12.3 of the earth vignette. It's because adding the variable has a deleterious effect on the model, as measured in terms of the model's estimated predictive power on new data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.