27
$\begingroup$

When building a CART model (specifically classification tree) using rpart (in R), it is often interesting to know what is the importance of the various variables introduced to the model.

Thus, my question is: What common measures exists for ranking/measuring variable importance of participating variables in a CART model? And how can this be computed using R (for example, when using the rpart package)

For example, here is some dummy code, created so you might show your solutions on it. This example is structured so that it is clear that variable x1 and x2 are "important" while (in some sense) x1 is more important then x2 (since x1 should apply to more cases, thus make more influence on the structure of the data, then x2).

set.seed(31431)
n <- 400
x1 <- rnorm(n)
x2 <- rnorm(n)
x3 <- rnorm(n)
x4 <- rnorm(n)
x5 <- rnorm(n)

X <- data.frame(x1,x2,x3,x4,x5)

y <- sample(letters[1:4], n, T)
y <- ifelse(X[,2] < -1 , "b", y)
y <- ifelse(X[,1] < 0 , "a", y)

require(rpart)
fit <- rpart(y~., X)
plot(fit); text(fit)

info.gain.rpart(fit) # your function - telling us on each variable how important it is

(references are always welcomed)

$\endgroup$
  • $\begingroup$ how does this question differ from stats.stackexchange.com/questions/5443/… ? $\endgroup$ – steffen Jan 24 '11 at 9:14
  • $\begingroup$ That question refers to knowing which predictor was relevant for a particular categorical value of the dependent variable. This question is more broad (variable importance/ranking without detecting to which nominal value it effects). Since that question wasn't answered I thought it worth to phrase it in a more general way in the hopes that someone might be able to help ... $\endgroup$ – Tal Galili Jan 24 '11 at 9:19
42
$\begingroup$

Variable importance might generally be computed based on the corresponding reduction of predictive accuracy when the predictor of interest is removed (with a permutation technique, like in Random Forest) or some measure of decrease of node impurity, but see (1) for an overview of available methods. An obvious alternative to CART is RF of course (randomForest, but see also party). With RF, the Gini importance index is defined as the averaged Gini decrease in node impurities over all trees in the forest (it follows from the fact that the Gini impurity index for a given parent node is larger than the value of that measure for its two daughter nodes, see e.g. (2)).

I know that Carolin Strobl and coll. have contributed a lot of simulation and experimental studies on (conditional) variable importance in RFs and CARTs (e.g., (3-4), but there are many other ones, or her thesis, Statistical Issues in Machine Learning – Towards Reliable Split Selection and Variable Importance Measures).

To my knowledge, the caret package (5) only considers a loss function for the regression case (i.e., mean squared error). Maybe it will be added in the near future (anyway, an example with a classification case by k-NN is available in the on-line help for dotPlot).

However, Noel M O'Boyle seems to have some R code for Variable importance in CART.

References

  1. Sandri and Zuccolotto. A bias correction algorithm for the Gini variable importance measure in classification trees. 2008
  2. Izenman. Modern Multivariate Statistical Techniques. Springer 2008
  3. Strobl, Hothorn, and Zeilis. Party on!. R Journal 2009 1/2
  4. Strobl, Boulesteix, Kneib, Augustin, and Zeilis. Conditional variable importance for random forests. BMC Bioinformatics 2008, 9:307
  5. Kuhn. Building Predictive Models in R Using the caret Package. JSS 2008 28(5)
$\endgroup$
  • 1
    $\begingroup$ Really - deserves far more votes than it has. $\endgroup$ – Matt Parker Jan 26 '11 at 17:21
  • $\begingroup$ +1 for the great answer. And update for late-comers (like myself) importance() in randomForest does individual variable importance with mean deacrease in acccuracy and mean gini decrease. $\endgroup$ – Zhubarb Sep 28 '15 at 16:48
3
$\begingroup$

The following function(from Caret package) can be used for evaluating variable importance in rpart trees. I corrected a bug in the Caret function when this only root node in the tree.

varImp <- function(object, surrogates = FALSE, competes = TRUE, ...)
  {
tmp <- rownames(object$splits)

 allVars <- colnames(attributes(object$terms)$factors)
if(is.null(tmp))
  {
  out<-NULL
    zeros <- data.frame(x = rep(0, length(allVars)),
                        Variable = allVars)
    out <- rbind(out, zeros)
  }

else {

rownames(object$splits) <- 1:nrow(object$splits)
splits <- data.frame(object$splits)
    splits$var <- tmp
splits$type <- ""

frame <- as.data.frame(object$frame)
    index <- 0
    for(i in 1:nrow(frame))
      {
        if(frame$var[i] != "<leaf>")
          {
            index <- index + 1
            splits$type[index] <- "primary"
            if(frame$ncompete[i] > 0)
              {
                for(j in 1:frame$ncompete[i])
                  {
                    index <- index + 1
                    splits$type[index] <- "competing"
                  }
              }
            if(frame$nsurrogate[i] > 0)
              {
                for(j in 1:frame$nsurrogate[i])
                  {
                    index <- index + 1
                    splits$type[index] <- "surrogate"
                  }
              }
          }
      }
    splits$var <- factor(as.character(splits$var))
    if(!surrogates) splits <- subset(splits, type != "surrogate")
    if(!competes) splits <- subset(splits, type != "competing")
    out <- aggregate(splits$improve,
                 list(Variable = splits$var),
                 sum,
                 na.rm = TRUE)

allVars <- colnames(attributes(object$terms)$factors)
if(!all(allVars %in% out$Variable))
      {
        missingVars <- allVars[!(allVars %in% out$Variable)]
        zeros <- data.frame(x = rep(0, length(missingVars)),
                            Variable = missingVars)
        out <- rbind(out, zeros)
      }
    }
    out2 <- data.frame(Overall = out$x)
rownames(out2) <- out$Variable
out2

}

The following r code will produce importance scores for a rpart tree "fit"

 varImp(fit)
$\endgroup$
  • $\begingroup$ Thanks. Did you report the bug to Max? (the maintainer of the caret package) $\endgroup$ – Tal Galili Oct 14 '14 at 19:11
1
$\begingroup$

I think chl has pretty much answered the first part:

What common measures exists for ranking/measuring variable importance of participating variables in a CART model?

With respect to the second part of your question:

And how can this be computed using R (for example, when using the rpart package)

You can find the variable importance using rpart by using summary(fit). This outputs the variable importance among several other things. You can read more about it here: https://cran.r-project.org/web/packages/rpart/rpart.pdf. Refer page 25.

$\endgroup$
0
$\begingroup$

names(result) shows variable.importance

result$variable.importance should help?

$\endgroup$
  • 3
    $\begingroup$ I believe the question has more to do with the advantages or popularity of some measures of variable importance than how to print the ones available in R for a particular method. $\endgroup$ – chl Mar 26 '13 at 11:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.