# Recursive feature elimination (rfe) performs poorly with binary outcome

Backward elimination with random forests does not work as expected in a simple test case with binary outcome and three continuous predictors.

Below, I generate a binary outcome based on a single predictor, X2. The other two predictors, X1 and X3 are noise. I then use backward elimination via the rfe function in caret with random forests via package randomForest. The rfe algorithm consistently fails to isolate X2 as the only important predictor. Of course, glm has no problem getting it right because the model is correctly specified. But given the importance rankings from randomForest, which clearly favor X2, I was expecting better accuracy from rfe.

Any ideas for how to improve performance with rfe?

library(caret)
library(randomForest)

rfDGP <- function(n = 500) {
X1 <- rnorm(n); X2 <- rnorm(n); X3 <- rnorm(n)
ps <- (1 + exp( -(0 + X2) ) )^-1
T <- rbinom(n = n, size = 1, prob = ps)
dat <- data.frame(cbind(X1, X2, X3, T))
dat$T <- factor(dat$T)
dat
}

# Generate data
set.seed(12345)
dat <- rfDGP()

# Use glm
summary(glm(T ~ ., family = "binomial", data = dat))

# Examine importance from randomForest()
vi <- randomForest(x = dat[,paste0("X", 1:3)], y = dat$T, importance = TRUE) importance(vi) # Use caret::rfe rfFuncs_acc <- rfFuncs ctrl_acc <- rfeControl(method = "cv", number = 10, functions = rfFuncs_acc, verbose = FALSE) (out_acc <- rfe(x = dat[,paste0("X", 1:3)], y = dat$T,
sizes = 1:3,
metric = "Accuracy",
rfeControl = ctrl_acc,
ntree = 500)$optVariables)  It's not the seed. I've tried 1, 12, 123, 1234, and 12345 and none gets it right. It's also not the use of "Accuracy". I've used "Kappa", and "ROC" and neither helps (see below). (out_kap <- rfe(x = dat[,paste0("X", 1:3)], y = dat$T,
sizes = 1:3,
metric = "Kappa",
rfeControl = ctrl_acc,
ntree = 500)$optVariables) rfFuncs_auc <- rfFuncs rfFuncs_auc$summary <- twoClassSummary
ctrl_auc <- rfeControl(method = "cv", number = 10, functions = rfFuncs_auc,
verbose = FALSE)
(out_auc <- rfe(x = dat[,paste0("X", 1:3)],
y = dat$T, sizes = 1:3, metric = "ROC", rfeControl = ctrl_auc, ntree = 500)$optVariables)