Can I see the contribution way of an input variable in random forest model?

Question

I have a randomforest model with 10 inputs to predict 1 output in R.

Let's say I ran the following code (from http://machinelearningmastery.com/feature-selection-with-the-caret-r-package)

# ensure results are repeatable
set.seed(7)
# load the library
library(mlbench)
library(caret)
# load the dataset
data(PimaIndiansDiabetes)
# prepare training scheme
control <- trainControl(method="repeatedcv", number=10, repeats=3)
# train the model
model <- train(diabetes~., data=PimaIndiansDiabetes, method="lvq", preProcess="scale", trControl=control)
# estimate variable importance
importance <- varImp(model, scale=FALSE)
# summarize importance
print(importance)
# plot importance
plot(importance)

Now I can see the importance of each variable in to the performance of the model, but can I know the influence way? More precisely, for instance, for variable 'insulin', does it mean more insulin will lead to more chance of diabetes, or vice versa? How can I know it from randomForest model?

Thank you,

Answer 1

In general, random forest is not an additive model (see http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Sixth%20Printing.pdf chapter 8 for a special case when it is) so the influence of a single variable can't be properly assessed independently of the others. You might try, for example, the Generalized Additive Models (GAM) approach if you want to separately describe influence of every predictor, but the model's performance might not be as good as with RF.

However, you might fix the other predictors on some selected levels, and - using e.g. the predict function - see how insulin affects the risk of diabetes "conditionally" on those other variables. By considering several different combinations of predictors other than insulin, you might get some approximate knowledge about insulin's influence and its interaction with other variables.

Answer 2

I have worked a year to write a package to make users learn the structure of RF models (I'd like to support other decision tree ensembles as well). My package forestFloor make use of feature contributions, that very crudely could be called "2. generation partial dependence plot". Maybe the also cool package iceBox would be 2. gen and forestFloor would be something slightly different. One could also check out rminer or edarf. The advantages of forestFloor compared to PD plots are:

• It is easier to identify hidden interactions
• The explained variance of prediction of strictly additive interpretation of random forest is quantifiable. Also 2nd or higher order interpretations is quantifiable.
• Quite fast to compute
• Multi classification is also supported
• There is useful probabilistic interpretation of classification models'*'

'*' (assuming the probabilistic predictions of the RF model is fairly calibrated, and they often are)

How to explore a Pima_indians_diabetis RF model: Ok, you can make it easier for yourself and not explore a overly complex model structure. Choose to train the most regularized RF model with maximum (or near maximum) cross-validated predictive performance. When the model do not explain target much, the learned structure cannot have as many reproducible details. Lowering mtry tend to use all features more evenly and increase tree decorrelation. Lowering sampsize tend reduce the amount of interaction in the model structure and make the model structure more smooth + even more tree decorrelation. Some recommend to prune ensemble trees or limit tree depth. These parameters will smooth model structure also, but without the extra tree decorrelation. Thus, just stick to sampsize.

library(mlbench)
library(randomForest)
library(forestFloor)
library(AUC)

data(PimaIndiansDiabetes)
y = PimaIndiansDiabetes$diabetes
X = PimaIndiansDiabetes
X = X[,!names(X)=="diabetes"]
rf.default = randomForest(X,y,ntree=5000)
rf.robust  = randomForest(X,y,sampsize=25,ntree=5000,mtry=4,
 keep.inbag = T,keep.forest = T) #need for forestFloor

#plot crosvalidated predictive performance
plot(roc(rf.default$votes[,2],y),main="ROC: default black, robust is red")

plot(roc(rf.robust$votes[,2],y),col=2,add = T)

The rf.robust and rf.default seem to predict equally good.

print(auc(roc(rf.default$votes[,2],y)))
0.8264776

print(auc(roc(rf.robust$votes[,2],y)))
0.8290821

computing feature contributions and visualization

Feature contributions are computed for each feature for each sample. Feature contributions summarize the change of predicted probability a given sample gets when splitted by a given variable. Each sample is plotted by each variable. The y-axis can be understood as the change of probability a given sample got, due to splits by a given variable. The x-axis the variable value it self. Variable plots are sorted by variable importance(in reading direction). It can be observed that the most important variables inflict the highest changes of probability. The blood glucose>175 will yield +18-25% probability according to the model. A color gradient by glucose colors all plots. Hereby it is easy to see that all other variables slightly interact with glocuse. The most important variable will likely perform more splits with high interleaf variance, and thus are other variables biased to mainly interact with glucose downstream in the dicision trees (this effect reduced with mtry=1). All variables tend to interact in a way which reduce the predicted effect of glucose level. This interaction effect is a normal bias of randomForest model and can be observed in many other training sets. I would not think too much about what such an learned interaction, relates to the underlying causal mechanism of diabetis. A linear fit quantifies for each variable how well the variable effect can be understood as a strictly additive effect. R2 = 1 - residual.variance/total.variance. Feature contributions of most variables are explained >90% by the variable value it self. Thus these 2D visualization are not a bad approximation of the full RF model structure.

ff = forestFloor(rf.robust,X,binary_reg = T,calc_np=T)
Col = fcol(ff,cols=1,outlier.lim = 2.5)
plot(ff,col=Col,plot_GOF = T)

Sometimes it is important to understand the effect of variable in a broader context than by the variable itself. The color gradient help to find a suitable context. Often the most important variable is the best context to include in a visualization. Pressure is variable effect which is the least explained by itself, also the interaction pattern with glucose is "positive" in contrary to all other interactions. Thus RF model has small interaction that high glucose and high pressure at the same time increase the risk of diabetes even more.

Remember missing pressure was assigned the value zero. Feature contributions of pressure is plotted against pressure value and glucose value. Fitted surface(grey) estimate explained variance. A clear interaction effect can be observed.

show3d(ff,c(1,5),5,col=Col,plot_GOF = T) library(rgl); rgl.snapshot("3dPressure.png")

slides from forestFloor presentation at useR 2015 Aalborg