2
$\begingroup$

Backward elimination with random forests does not work as expected in a simple test case with binary outcome and three continuous predictors.

Below, I generate a binary outcome based on a single predictor, X2. The other two predictors, X1 and X3 are noise. I then use backward elimination via the rfe function in caret with random forests via package randomForest. The rfe algorithm consistently fails to isolate X2 as the only important predictor. Of course, glm has no problem getting it right because the model is correctly specified. But given the importance rankings from randomForest, which clearly favor X2, I was expecting better accuracy from rfe.

Any ideas for how to improve performance with rfe?

library(caret)  
library(randomForest)  

rfDGP <- function(n = 500) {  
  X1 <- rnorm(n); X2 <- rnorm(n); X3 <- rnorm(n)  
  ps <- (1 + exp( -(0 + X2) ) )^-1 
  T <- rbinom(n = n, size = 1, prob = ps)
  dat <- data.frame(cbind(X1, X2, X3, T))
  dat$T <- factor(dat$T)
  dat
}

# Generate data
set.seed(12345)
dat <- rfDGP()

# Use glm
summary(glm(T ~ ., family = "binomial", data = dat))

# Examine importance from randomForest()
vi <- randomForest(x = dat[,paste0("X", 1:3)], y = dat$T, importance = TRUE)
importance(vi)

# Use caret::rfe
rfFuncs_acc <- rfFuncs
ctrl_acc <- rfeControl(method = "cv", number = 10, functions = rfFuncs_acc, 
                       verbose = FALSE)
(out_acc <- rfe(x = dat[,paste0("X", 1:3)], 
                y = dat$T,
                sizes = 1:3,
                metric = "Accuracy",
                rfeControl = ctrl_acc,
                ntree = 500)$optVariables)

It's not the seed. I've tried 1, 12, 123, 1234, and 12345 and none gets it right. It's also not the use of "Accuracy". I've used "Kappa", and "ROC" and neither helps (see below).

(out_kap <- rfe(x = dat[,paste0("X", 1:3)], 
                y = dat$T,
                sizes = 1:3,
                metric = "Kappa",
                rfeControl = ctrl_acc,
                ntree = 500)$optVariables)

rfFuncs_auc <- rfFuncs
rfFuncs_auc$summary <- twoClassSummary
ctrl_auc <- rfeControl(method = "cv", number = 10, functions = rfFuncs_auc, 
                       verbose = FALSE)
(out_auc <- rfe(x = dat[,paste0("X", 1:3)], 
                y = dat$T,
                sizes = 1:3,
                metric = "ROC",
                rfeControl = ctrl_auc,
                ntree = 500)$optVariables)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.