R: Importance of Categorical Variables in Random Forests [closed]

Question

I'm applying a random forest algorithm, using the randomForest library in R, on a data set with 3 variables (gre, gpa, rank), one of the variables (rank) is categorical with 4 levels (1, 2, 3, 4), when extracting feature importance I get the "Mean Decrease Gini" of each variable, my question is: Is there a way to get the importance of each level of the rank predictor, the influence of each one of'em on the dependent variable, the significance of each or something similar to these three in R when using a randomForest classifier or any classifier for that matter?

The code:

library(randomForest)

mydata <- read.csv("https://stats.idre.ucla.edu/stat/data/binary.csv")
mydata$rank <- factor(mydata$rank)

RF <- randomForest(factor(admit) ~gre + gpa + rank, data = mydata)
importance(RF)

output:

A matrix: 3 × 1 of type dbl     
    MeanDecreaseGini
gre 30.63603
gpa 42.76386
rank 18.85539

Thanks in advance.

Answer 1

You can't really get back the contributions of each variable because the categorical column is encoded as one column (unlike linear regression) and this is used as one whole variable, you can see more in this answer.

If you need to get some kind of estimate, say for a publication, you can try something like one hot encoding, and pass it to randomForest, below I simply used the model matrix.

set.seed(111)
X = model.matrix(~0+gre+gpa+rank,data=mydata) 
RF = randomForest(y=factor(mydata$admit),x=X)

importance(RF)
      MeanDecreaseGini
gre          21.995150
gpa          30.902523
rank1         6.720115
rank2         3.320136
rank3         3.001383
rank4         3.018972

However, you can see now the importance of the other variables slightly changes because it is a different model now. You can check the predictions against the other formula

RF_formula <- randomForest(factor(admit) ~gre + gpa + rank, data = mydata)
plot(predict(RF,type="prob"),
predict(RF_formula,type="prob"),xlab="onehot",ylab="cat")

It's not that dramatically different, bottom-line is, think about it's interpretation. you can also read more in this discussion .

In the event, you have more than one categorical variable, you can do:

library(caret)
X = predict(dummyVars(~ gre+gpa+rank,data=mydata),mydata)