1
$\begingroup$

I'm using random forests in R for classification problems (via package randomForest). I have two objectives: to get the classification accuracy, and to find out which variables contribute most to the discrimination. My question is about the latter: how to find a consistent way to judge which variables are important.

The importance scores can be extracted either raw or divided by the SD, but in either case they are correlated with the numbers of trees in the forest and with the number of observations in the dataset. It occurred to me that the problem was analogous to choosing the number of components from a PCA, so I have been experimenting with using a broken stick model. Here is my example:

library(randomForest)
library(vegan)
data("imports85")

#Run a random forest
set.seed(47)
rf<-randomForest(imports85[,10:18], imports85$bodyStyle, 
                 importance = T, ntree=5000, na.action = na.exclude)
rf

imp<-importance(rf)[,"MeanDecreaseAccuracy"]

#Broken stick model where 'total variance' = sum of importance values > 0
stick<-bstick(length(imp), sum(imp[imp>0]))
stick<-stick[order(stick, decreasing = T)]

#Plot the predicted importance and overlay the real data
scatter.smooth(seq(9,1)~stick, type = "n", 
               xlim = c(min(imp), max(imp)), xlab = "Importance", ylab = "Variable")
points(seq(9,1)~sort(imp, decreasing = T))

#Count how many variables had greater than predicted importance
sum(sort(imp, decreasing = T) > stick)

My question is whether this is a valid approach. It makes sense to me, as the broken stick model represents a null model to compare which variables are more important than would be expected by chance. Negative values are excluded when calculating 'total importance', as otherwise the broken stick model would underestimate the value of the most important variables. I would be very grateful for insights/expertise (please be gentle - I'm an ecologist rather than a mathematician)!

$\endgroup$

1 Answer 1

1
$\begingroup$

This seems like a sensible approach and has been used in other areas of changepoint detection to pick the penalty. See Lavielle (2005) for an intuitive approach (for changepoints rather than random forests) and the recent pre-print which uses the broken stick model.

Romano et. al (2020+) https://arxiv.org/abs/2005.01379

Lavielle (2005) https://rmgsc.cr.usgs.gov/outgoing/threshold_articles/Lavielle2005.pdf

I'm not sure if this has been used with Random Forests before as that isn't my area of research. But hopefully the knowledge that this approach is used, with success, in other areas with similar problems, is helpful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.