39
$\begingroup$

I've been playing around with random forests for regression and am having difficulty working out exactly what the two measures of importance mean, and how they should be interpreted.

The importance() function gives two values for each variable: %IncMSE and IncNodePurity. Is there simple interpretations for these 2 values?

For IncNodePurity in particular, is this simply the amount the RSS increase following the removal of that variable?

$\endgroup$
  • 1
    $\begingroup$ Have you looked at ?importance ? There's an explanation there on what both measures mean... $\endgroup$ – Nick Sabbe Jul 4 '11 at 8:30
  • 2
    $\begingroup$ @Nick Sabbe, I have, and am trying to wrap my head around them. I was wondering if there are any nice intuitive interpretations for them. $\endgroup$ – dcl Jul 4 '11 at 8:53
42
$\begingroup$

The first one can be 'interpreted' as follows: if a predictor is important in your current model, then assigning other values for that predictor randomly but 'realistically' (i.e.: permuting this predictor's values over your dataset), should have a negative influence on prediction, i.e.: using the same model to predict from data that is the same except for the one variable, should give worse predictions.

So, you take a predictive measure (MSE) with the original dataset and then with the 'permuted' dataset, and you compare them somehow. One way, particularly since we expect the original MSE to always be smaller, the difference can be taken. Finally, for making the values comparable over variables, these are scaled.

For the second one: at each split, you can calculate how much this split reduces node impurity (for regression trees, indeed, the difference between RSS before and after the split). This is summed over all splits for that variable, over all trees.

Note: a good read is Elements of Statistical Learning by Hastie, Tibshirani and Friedman...

$\endgroup$
  • 3
    $\begingroup$ Cheers, I actually have that book open now :) $\endgroup$ – dcl Jul 4 '11 at 9:24
  • $\begingroup$ What does RSS mean? $\endgroup$ – DavideChicco.it Oct 27 '16 at 21:53
  • $\begingroup$ RSS is Residual Sum of Squares $\endgroup$ – Barker Dec 6 '16 at 21:06
10
$\begingroup$

Random Forest importance metrics as implemented in the randomForest package in R have quirks in that correlated predictors get low importance values.

http://bioinformatics.oxfordjournals.org/content/early/2010/04/12/bioinformatics.btq134.full.pdf

I have a modified implementation of random forests out on CRAN which implements their approach of estimating empirical p values and false discovery rates, here

http://cran.r-project.org/web/packages/pRF/index.html

$\endgroup$
  • 1
    $\begingroup$ does this explain the different output of variable importance if you use randomForest with the caret package like caret::train(method="rf", importance = TRUE, ...) ?? $\endgroup$ – Agile Bean Dec 21 '18 at 8:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.