4
$\begingroup$

I am using a decision tree and random forest for a classification problem. The output is binary {0,1} and some of the input variables are categorical while the others are continuous.

I would like to know if it is possible to extract odds ratio from one of these models. Odds ratio similar to the ones that are commonly obtained using logistic regression.

I am using R with the package rpart and randomforest so the best answer would be a theoretical explanation and an R code.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
+50
$\begingroup$

You could use the idea of partial dependency plots which basically plot the change in the average predicted value (from a given model) as specific variable(s) vary over their marginal distribution. This means that you plot the average predicted value as the predictor(s) vary over their domain, averaging over the values of the other predictors with their values set as observed in the training data.

Here is a snippet from Elements of Statistical Learning

enter image description here

One could always use a brute force method to average the predicted value that results by "re-scoring" the same (training) data set repeatedly, each time only changing the value of the specific variable(s) of interest, leaving everything else as was observed. There are computational shortcuts in tree algorithms and randomForest in R contains the function partialPlot that makes this brute force method unnecessary.

In this example, the predicted probability for aspiration is estimated and the average value at the two levels of numOfDoors is obtained. In theory, you can do this for any predictor(s). Continuous variable effects are not linear in a random forest or decision tree generally, so you would have to create interesting discrete partitions of the variable to calculate the odds ratio. I notice how volatile the value is in this small data set, so you wont get the same result twice and wont match the value I have listed, but the process should be clear.

library(randomForest)
library(boot)
library(car)

data(imports85)
table(na.omit(imports85)$aspiration)
table(na.omit(imports85)$numOfDoors)

mod.rf<-randomForest(aspiration~numOfDoors+wheelBase+length+width+height, data=na.omit(imports85),ntree=1000,keep.forest=TRUE)

#partial dependence plot for numOfDoors (two or four)
#note: n.pt is useful for continuous variables
pdp<-boot::inv.logit(partialPlot(mod.rf,na.omit(imports85),numOfDoors)$y)

#odds ratio for aspiration = std

#odds two cyclinder
odds.2<- (pdp[2]/(1-pdp[2]))
odds.4<- (pdp[1]/(1-pdp[1]))

odds.2/odds.4  #1.74
Improve this answer
$\endgroup$
8
  • 1
    $\begingroup$ Very good answer. What is the cause of the volatility of the result? Is this due to the instability of the model mod.rf? Random Forest is usually supposed to be robust. $\endgroup$
    – Donbeo
    Apr 12, 2014 at 13:41
  • $\begingroup$ Sorry I am a bit confused. Once you have $\bar f(X_s)$ how do you compute the odds ratio? $\endgroup$
    – Donbeo
    Apr 12, 2014 at 13:46
  • $\begingroup$ random forests is by its nature random (bagging and random sampling of the predictors to attempt at each split). There may well be very correlated predictors with this toy example as well. $\endgroup$
    – B_Miner
    Apr 12, 2014 at 14:02
  • 1
    $\begingroup$ Once that estimate is obtained, use the inverse logit to transform the output to the probability scale (randomForest uses a logit link in the partial dependence plot). Then compute the odds ratio as normal with probabilities: pages.uoregon.edu/aarong/teaching/G4075_Outline/node15.html $\endgroup$
    – B_Miner
    Apr 12, 2014 at 14:05
  • 1
    $\begingroup$ Then its just the same but each of the odds ratios is done for a given level of the predictor relative to the reference level. This is all analogous to logistic regression. With a truly continuous variable like I mentioned, you could discretize the predictor into bins or simply look at the predicted probability at two different levels of the predictor (x and x+h). This is a non-linear function so it depends on the point x (i.e. it is not constant). $\endgroup$
    – B_Miner
    Apr 12, 2014 at 15:06
0
$\begingroup$

A couple points on this:

First, random forests and decisions trees don't naturally output a probability as the response. I'm sure there is some way to set them up to provide this but you'll have to take that into account when tuning your model.

Second, you may be aware of this but bear in mind that with a logistic model, the odds ratio for different values of a single variable (ignoring interactions) is constant regardless of values of the other variables. With random forests that will not be the case because the whole thing is built to actually find interactions. So when you express the odds ratio, it would have to be between two unique values of your dependent variable set. (B_Miner's response has some good thoughts about how to get an average odds ratio between values of a single variable, but that would be only an average.)

So bottom line is that you should be careful about how you're using the odds ratio here. If what you're after is an odds ratio over values of a single variable, a random forest is a very disadvantageous choice of model structure.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I agree that if an odds ratio is sought as the primary objective, it makes sense to stick with logistic regression, but of course all those assumptions have to be met. I disagree that a probability of class membership is not "naturally" output by tree based models though, these (and GBM and others) absolutely do. $\endgroup$
    – B_Miner
    Apr 12, 2014 at 13:55
  • $\begingroup$ Ya you're right about that. I really just meant to be sure to specify in whatever software that the output should be a probability, which is a bit of a trivial comment. It's been a few months since I've used these libraries in R, but I seem to recall the predict method giving a discrete outcome by default. $\endgroup$
    – Mike Nute
    Apr 13, 2014 at 2:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.