# Feature importance for random forest classification of a sample

Using a random forest is it possible to determine which features were the driving features to classify a specific sample as class A?

I know I can ask which features are more important to perform classification of ANY sample, but can I ask this for a specific sample? E.g. Why was sample 1 classified as A? Which of its features were much more like class A than class B?

Does it even make sense to ask this question of a random forest?

Bonus points on how to do it with sklearn in python :)

Variable importance accounts for the increase in out-of-bag cross-validated prediction error. It would be possible but not meaningful to account for the change of prediction error by one sample only. As one sample only can be correctly or wrongly predicted, such a term would be very unstable and crude.

You could check out 'local variable importance', 'partial dependence plots' or 'feature contributions'. Here's an example from my package forestFloor using feature contributions. Each plot shows the change of predicted class probability as function each variable. For the iris data set, there no strong variable interactions. Therefore, the model structure can be boiled down to a 2D visualization. The R-sqaured terms quantifies how much the model structure deviates from this main effect only interpretation/visualization.

library(forestFloor)
library(randomForest)
data(iris)
X = iris[,!names(iris) %in% "Species"]
Y = iris[,"Species"]

rf = randomForest(X,Y,
keep.forest=TRUE, #mandatory for classification
replace=FALSE,    #if TRUE use trimTrees::cinbag, not randomForest
keep.inbag=TRUE,  #mandatory always for forestFloor
sampsize =15 )    #optional:smaller trees smoother model structure

ff = forestFloor(rf.fit  = rf,           # mandatory
X       = X,            # mandatory
calc_np = "sad monkey", # this input takes no effect for classification
binary_reg = FALSE)     # can change two class classification to regression
# Thus cannot be TRUE for IRIS (three class)

plot(ff,plot_GOF=TRUE,cex=.7,
colLists=list(c("#FF0000A5"),
c("#00FF0050"),
c("#0000FF35")))


• Thanks for your description, example, and terminology. It's amazing how much easier it is to find information when you know the jargon :) – CHP Sep 27 '15 at 19:43
• For anyone interested, I found this implementation with some of these things for sklearn: blog.datadive.net/… It needs the dev version of sklearn (0.17) – CHP Sep 27 '15 at 21:50