I'm applying a random forest algorithm, using the randomForest library in R, on a data set with 3 variables (gre, gpa, rank), one of the variables (rank) is categorical with 4 levels (1, 2, 3, 4), when extracting feature importance I get the "Mean Decrease Gini" of each variable, my question is: Is there a way to get the importance of each level of the rank predictor, the influence of each one of'em on the dependent variable, the significance of each or something similar to these three in R when using a randomForest classifier or any classifier for that matter?

The code:


mydata <- read.csv("https://stats.idre.ucla.edu/stat/data/binary.csv")
mydata$rank <- factor(mydata$rank)

RF <- randomForest(factor(admit) ~gre + gpa + rank, data = mydata)


A matrix: 3 × 1 of type dbl     
gre 30.63603
gpa 42.76386
rank 18.85539

Thanks in advance.


1 Answer 1


You can't really get back the contributions of each variable because the categorical column is encoded as one column (unlike linear regression) and this is used as one whole variable, you can see more in this answer.

If you need to get some kind of estimate, say for a publication, you can try something like one hot encoding, and pass it to randomForest, below I simply used the model matrix.

X = model.matrix(~0+gre+gpa+rank,data=mydata) 
RF = randomForest(y=factor(mydata$admit),x=X)

gre          21.995150
gpa          30.902523
rank1         6.720115
rank2         3.320136
rank3         3.001383
rank4         3.018972

However, you can see now the importance of the other variables slightly changes because it is a different model now. You can check the predictions against the other formula

RF_formula <- randomForest(factor(admit) ~gre + gpa + rank, data = mydata)

enter image description here

It's not that dramatically different, bottom-line is, think about it's interpretation. you can also read more in this discussion .

In the event, you have more than one categorical variable, you can do:

X = predict(dummyVars(~ gre+gpa+rank,data=mydata),mydata)
  • $\begingroup$ Thank you @StupidWolf, great answer. If you don't mind me asking, what does this line of code: "X = model.matrix(~0+gre+gpa+rank,data=mydata)" do? and what does "0" stand for? Thanks in advance. $\endgroup$ Apr 1, 2020 at 22:04
  • 1
    $\begingroup$ of course, in this example it means a model without intercept. Normally when you fit a linear model, it will take one of the factor / categorical terms as reference, and make binary variables out of the other factors.. setting 0 when there is only 1 categorical will ensure all factors are encoded 0/1 $\endgroup$
    – StupidWolf
    Apr 1, 2020 at 22:08
  • 2
    $\begingroup$ if you have 2 or more categorical columns, you might have to use dummyVars from caret rdocumentation.org/packages/caret/versions/6.0-85/topics/… to do the onehot encoding $\endgroup$
    – StupidWolf
    Apr 1, 2020 at 22:10
  • $\begingroup$ Thank you very much for the help @StupidWolf, your second comment provided me with the answer to a question I was about to ask :) One last question: should I set the intercept to 0 when I have 2 or more categorical variable or is that only the case for when we have a single categorical variable? $\endgroup$ Apr 1, 2020 at 22:16
  • 1
    $\begingroup$ That only works when you have a single categorical variable.. I can update my answer to include use of dummyVars from caret, which works for more than 1 $\endgroup$
    – StupidWolf
    Apr 1, 2020 at 22:18

Not the answer you're looking for? Browse other questions tagged or ask your own question.