4
$\begingroup$

I am using a decision tree and random forest for a classification problem. The output is binary {0,1} and some of the input variables are categorical while the others are continuous.

I would like to know if it is possible to extract odds ratio from one of these models. Odds ratio similar to the ones that are commonly obtained using logistic regression.

I am using R with the package rpart and randomforest so the best answer would be a theoretical explanation and an R code.

$\endgroup$
3
+50
$\begingroup$

You could use the idea of partial dependency plots which basically plot the change in the average predicted value (from a given model) as specific variable(s) vary over their marginal distribution. This means that you plot the average predicted value as the predictor(s) vary over their domain, averaging over the values of the other predictors with their values set as observed in the training data.

Here is a snippet from Elements of Statistical Learning

enter image description here

One could always use a brute force method to average the predicted value that results by "re-scoring" the same (training) data set repeatedly, each time only changing the value of the specific variable(s) of interest, leaving everything else as was observed. There are computational shortcuts in tree algorithms and randomForest in R contains the function partialPlot that makes this brute force method unnecessary.

In this example, the predicted probability for aspiration is estimated and the average value at the two levels of numOfDoors is obtained. In theory, you can do this for any predictor(s). Continuous variable effects are not linear in a random forest or decision tree generally, so you would have to create interesting discrete partitions of the variable to calculate the odds ratio. I notice how volatile the value is in this small data set, so you wont get the same result twice and wont match the value I have listed, but the process should be clear.

library(randomForest)
library(boot)
library(car)

data(imports85)
table(na.omit(imports85)$aspiration)
table(na.omit(imports85)$numOfDoors)

mod.rf<-randomForest(aspiration~numOfDoors+wheelBase+length+width+height, data=na.omit(imports85),ntree=1000,keep.forest=TRUE)

#partial dependence plot for numOfDoors (two or four)
#note: n.pt is useful for continuous variables
pdp<-boot::inv.logit(partialPlot(mod.rf,na.omit(imports85),numOfDoors)$y)

#odds ratio for aspiration = std

#odds two cyclinder
odds.2<- (pdp[2]/(1-pdp[2]))
odds.4<- (pdp[1]/(1-pdp[1]))

odds.2/odds.4  #1.74
$\endgroup$
  • 1
    $\begingroup$ Very good answer. What is the cause of the volatility of the result? Is this due to the instability of the model mod.rf? Random Forest is usually supposed to be robust. $\endgroup$ – Donbeo Apr 12 '14 at 13:41
  • $\begingroup$ Sorry I am a bit confused. Once you have $\bar f(X_s)$ how do you compute the odds ratio? $\endgroup$ – Donbeo Apr 12 '14 at 13:46
  • $\begingroup$ random forests is by its nature random (bagging and random sampling of the predictors to attempt at each split). There may well be very correlated predictors with this toy example as well. $\endgroup$ – B_Miner Apr 12 '14 at 14:02
  • 1
    $\begingroup$ Once that estimate is obtained, use the inverse logit to transform the output to the probability scale (randomForest uses a logit link in the partial dependence plot). Then compute the odds ratio as normal with probabilities: pages.uoregon.edu/aarong/teaching/G4075_Outline/node15.html $\endgroup$ – B_Miner Apr 12 '14 at 14:05
  • 1
    $\begingroup$ Then its just the same but each of the odds ratios is done for a given level of the predictor relative to the reference level. This is all analogous to logistic regression. With a truly continuous variable like I mentioned, you could discretize the predictor into bins or simply look at the predicted probability at two different levels of the predictor (x and x+h). This is a non-linear function so it depends on the point x (i.e. it is not constant). $\endgroup$ – B_Miner Apr 12 '14 at 15:06
0
$\begingroup$

A couple points on this:

First, random forests and decisions trees don't naturally output a probability as the response. I'm sure there is some way to set them up to provide this but you'll have to take that into account when tuning your model.

Second, you may be aware of this but bear in mind that with a logistic model, the odds ratio for different values of a single variable (ignoring interactions) is constant regardless of values of the other variables. With random forests that will not be the case because the whole thing is built to actually find interactions. So when you express the odds ratio, it would have to be between two unique values of your dependent variable set. (B_Miner's response has some good thoughts about how to get an average odds ratio between values of a single variable, but that would be only an average.)

So bottom line is that you should be careful about how you're using the odds ratio here. If what you're after is an odds ratio over values of a single variable, a random forest is a very disadvantageous choice of model structure.

$\endgroup$
  • 1
    $\begingroup$ I agree that if an odds ratio is sought as the primary objective, it makes sense to stick with logistic regression, but of course all those assumptions have to be met. I disagree that a probability of class membership is not "naturally" output by tree based models though, these (and GBM and others) absolutely do. $\endgroup$ – B_Miner Apr 12 '14 at 13:55
  • $\begingroup$ Ya you're right about that. I really just meant to be sure to specify in whatever software that the output should be a probability, which is a bit of a trivial comment. It's been a few months since I've used these libraries in R, but I seem to recall the predict method giving a discrete outcome by default. $\endgroup$ – Mike Nute Apr 13 '14 at 2:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.