# Obtaining knowledge from a random forest

Random forests are considered to be black-boxes, but recently I was thinking what knowledge can be obtained from a random forest?

The most obvious thing is the importance of the variables, in the simplest variant it can be done just by calculating number of occurrences of a variable.
The second thing I was thinking are interactions. I think that if the number of trees is sufficiently large then number of occurrences of pairs of variables can be tested (something like chi square independence). The third thing are nonlinearities of variables. My first idea was just to look at a chart of a variable Vs score, but I'm not sure yet whether it makes any sense.

This things are probably well studied (just an intuition). I would grateful if anyone could point me how to examine those things properly.

Motivation

I want to use this knowledge to improve a logit model. I think (or at least I hope) that it is possible to find interactions and nonlinearities that were overlooked.

-

Random Forests are hardly a black box. They are based on decision trees, which are very easy to interpret:

#Setup a binary classification problem
require(randomForest)
data(iris)
set.seed(1)
dat <- iris
dat$Species <- factor(ifelse(dat$Species=='virginica','virginica','other'))
trainrows <- runif(nrow(dat)) > 0.3
train <- dat[trainrows,]
test <- dat[!trainrows,]

#Build a decision tree
require(rpart)
model.rpart <- rpart(Species~., train)


This results in a simple decision tree:

> model.rpart
n= 111

node), split, n, loss, yval, (yprob)
* denotes terminal node

1) root 111 35 other (0.68468468 0.31531532)
2) Petal.Length< 4.95 77  3 other (0.96103896 0.03896104) *
3) Petal.Length>=4.95 34  2 virginica (0.05882353 0.94117647) *


If Petal.Length < 4.95, this tree classifies the observation as "other." If it's greater than 4.95, it classifies the observation as "virginica." A random forest is simple a collection of many such trees, where each one is trained on a random subset of the data. Each tree then "votes" on the final classification of each observation.

model.rf <- randomForest(Species~., train, ntree=25, proximity=TRUE, importance=TRUE, nodesize=5)
> getTree(model.rf, k=1, labelVar=TRUE)
left daughter right daughter    split var split point status prediction
1             2              3  Petal.Width        1.70      1       <NA>
2             4              5 Petal.Length        4.95      1       <NA>
3             6              7 Petal.Length        4.95      1       <NA>
4             0              0         <NA>        0.00     -1      other
5             0              0         <NA>        0.00     -1  virginica
6             0              0         <NA>        0.00     -1      other
7             0              0         <NA>        0.00     -1  virginica


You can even pull out individual trees from the rf, and look at their structure. The format is slightly different than for rpart models, but you could inspect each tree if you wanted and see how it's modeling the data.

Furthermore, no model is truly a black box, because you can examine predicted responses vs actual responses for each variable in the dataset. This is a good idea regardless of what sort of model you are building:

library(ggplot2)
pSpecies <- predict(model.rf,test,'vote')[,2]
plotData <- lapply(names(test[,1:4]), function(x){
out <- data.frame(
var = x,
type = c(rep('Actual',nrow(test)),rep('Predicted',nrow(test))),
value = c(test[,x],test[,x]),
species = c(as.numeric(test$Species)-1,pSpecies) ) out$value <- out$value-min(out$value) #Normalize to [0,1]
out$value <- out$value/max(out$value) out }) plotData <- do.call(rbind,plotData) qplot(value, species, data=plotData, facets = type ~ var, geom='smooth', span = 0.5)  I've normalized the variables (sepal and petal length and width) to a 0-1 range. The response is also 0-1, where 0 is other and 1 is virginica. As you can see the random forest is a good model, even on the test set. Additionally, a random forest will compute various measure of variable importance, which can be very informative: > importance(model.rf, type=1) MeanDecreaseAccuracy Sepal.Length 0.28567162 Sepal.Width -0.08584199 Petal.Length 0.64705819 Petal.Width 0.58176828  This table represents how much removing each variable reduces the accuracy of the model. Finally, there are many other plots you can make from a random forest model, to view what's going on in the black box: plot(model.rf) plot(margin(model.rf)) MDSplot(model.rf, iris$Species, k=5)
plot(outlier(model.rf), type="h", col=c("red", "green", "blue")[as.numeric(dat\$Species)])


You can view the help files for each of these functions to get a better idea of what they display.

-
Thanks for the answer, there is a lot of useful info, but it is not exactly what I was looking for. Maybe I need to clarify better the motivation that is behind this question. I want to use a random forest to improve a logit model, by improve I mean to add some interaction or to use a nonlinear transformation. –  Tomek Tarczynski Jan 23 '12 at 10:47
@TomekTarczynski that's an interesting problem and similar to one I'm dealing with right now. I assume by "logit model" you mean logistic regression or something similar? I'm using lasso logistic regression (from the glmnet R package) to select predictors from a model with interactions between all pairs of variables. I haven't added in any nonlinear terms yet--but in principle that should be possible too. The only issue I guess is deciding what nonlinear terms to try (polynomial terms, exponential transforms, etc?). Also, I'm not picking up any higher-order interactions but that's easy too. –  Anne Z. Jan 25 '12 at 13:23
@Tomek, what are you not getting from this answer? If you are using the randomForest package in R then the plots Zach describes should be very useful. Specifically, you could use varImpPlot for feature selection in your logit model and partialPlot to estimate the type of transformation to try on continuous predictors in the logit model. I would suggest that the latter plot be used to determine where nonlinear relationships between predictor and response exists and then allows you to make that transformation explicitly or to use a spline on that variable. –  B_Miner Jan 25 '12 at 14:14
@b_miner - just a guess, but it sounds like tomek is asking how to find non-linear interactions between variables because logistic regression already captures the linear relationships. –  rm999 Jan 25 '12 at 15:04
@rm999 How do you define a non linear interaction in a logit model? Interaction terms created between transformed variables? –  B_Miner Jan 25 '12 at 15:19

To supplement these fine responses, I would mention use of gradient boosted trees (e.g. the GBM Package in R). In R, I prefer this to random forests because missing values are allowed as compared to randomForest where imputation is required. Variable importance and partial plots are available (as in randomForest) to aid in feature selection and nonlinear transformation exploration in your logit model. Further, variable interaction is addressed with Friedman’s H-statistic (interact.gbm) with reference given as J.H. Friedman and B.E. Popescu (2005). “Predictive Learning via Rule Ensembles.” Section 8.1. A commercial version called TreeNet is available from Salford Systems and this video presentation speaks to their take on variable interaction estimation Video.

-
I agree, GBMs are a logical next step from random forests. –  Zach Jan 25 '12 at 14:28
@B_miner: Great! I don't know how, but I have overlooked GBM. It seems that using GBM it is easy do detect interactions and nonlinearities. –  Tomek Tarczynski Jan 26 '12 at 7:55

I'm very interested in these type of questions myself. I do think there is a lot of information we can get out of a random forest.

About Interactions, it seems like Breiman and Cultier have already tried to look at it, especially for classification RFs.

To my knowledge, this has not been implemented in the randomForest R package. Maybe because it might not be as simple and because the meaning of "variable interactions" is very dependent of your problem.

About the nonlinearity, I'm not sure what you are looking for, regression forest are used for nonlinear multiple regression problems without any priors on what type of nonlinear function to use.

-

Using R you can produce a Dotchart of variable importance as measured by a Random Forest.

-